2GB file size limit? | Joomla Support | Forum Archive

The free forums are no longer in use. It remains available as read-only archive.

Avatar
Lost password?
Advanced Search
Forum Scope


Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
The forums are currently locked and only available for read only access
sp_TopicIcon
2GB file size limit?
January 12, 2011
8:04 pm
Avatar
Marcus
Member
Members
Forum Posts: 20
Member Since:
January 12, 2011
sp_UserOfflineSmall Offline

Hi!

I've been trying out Xcloner on my website hosted on a IIS server (my own locally hosted Win2008 R2 server). My website is quite large, 2,6GB in file size. I run Xcloner with default values and everything checks out fine, except for when the file size counter during backup exceeds 2GB, then the counter flips around and says a negative -2147393024. I watch the increasing file size of the backup file and it stops expanding at the 2GB limit although the Xcloner scripts continues and finally finishes successfully. The properties of the finished backup file is 2 147 569 664 bytes. The restore procedure doesn't work either, with an incomplete TAR file it just extracts two small folders. Note that this is on NTFS files system. I've checked the folder permissions, I have adjusted the max file size limit in php.ini and also the execution time limit.

So, I've tried to find info about any size limit, either a TAR size limit, PHP limit, or an Xcloner limit. Is there such a thing and if so, how can I get around the problem?

Thanks!

//Marcus

January 12, 2011
8:25 pm
Avatar
Ovidiu Liuta
Admin
Forum Posts: 2484
Member Since:
September 26, 2010
sp_UserOfflineSmall Offline

Hi Marcus!

XCloner does not have a file size limitation, we have successfully created backups over 4GB, however this 2GB limitation might come from your server filesystem, check with your hosting support if they allow files larger than 2GB to be created on the server. The other issue regarding the file size display is caused by the http://php.net/manua..... function, i will look into getting that fixed, however, please note, most php installs can't display file sizes over 4GB, so you might need to use another way of checking the correct file size of larger files.

Make sure also you have enough disk space on your system to store that backup. If you are unable to create backups over 2GB, then simply try and create smaller backups(under 2GB) by excluding some folders.

 

Ovidiu

January 12, 2011
9:10 pm
Avatar
Marcus
Member
Members
Forum Posts: 20
Member Since:
January 12, 2011
sp_UserOfflineSmall Offline

Thanks for the quick reply, Ovidiu!

The server I run my website on is located in the room next door so I manage it all by myself and I have installed it myself, running for two straight years now. There is no file size limit on the file system, since NTFS in Windows Server 2008 can support files >2GB. I have 15GB remaining of free disk space on the website disk volume, and the operating system volume has 5GB of free space. It should be sufficient for any temporary files created during backup.

I tried the compression alternative in the settings for Xcloner, but there was a time-out error and the backup file size never exceeded 1MB..

Since I'm on a Windows platform there isn't much chance of using anything but the TAR-compression like ZIP, is there? If I install WinZip or WinRAR, can I use that with Xcloner? It would be interesting to try to see if it can solve the problem in some way.

Thanks for your suggestions, I'll experiment a bit more. More feedback is welcome, of course ! Cool

 

//Marcus

January 12, 2011
9:29 pm
Avatar
Ovidiu Liuta
Admin
Forum Posts: 2484
Member Since:
September 26, 2010
sp_UserOfflineSmall Offline

Could you make a test and see if you can create a file larger than 2GB inside your backups diirectory? If you have fsutil on your server, follow this manual http://www.microsoft.....=true and launch this command(replace the PathName and length with some actual values)

 

fsutil file createnew PathName length

 

XCloner does not support the ZIP option(anymore...), in only supports the TAR GZIP option, and since you are on windows, you could try and use the TAR for Windows server utility http://gnuwin32.sour.....gtar.htm , but please note, for this you also need to have the exec() function enabled in your php config.

 

If you plan on trying the php TAR compress mode (TGZ), i would advise increasing the php.ini memory_limit and max_execution_time to larger values to avoid the timeout you get there, it might help.

 

Ovidiu

January 12, 2011
9:55 pm
Avatar
Marcus
Member
Members
Forum Posts: 20
Member Since:
January 12, 2011
sp_UserOfflineSmall Offline

I ran the command fsutil file createnew E:test.txt 4294967296 and the file was created successfully, so no problems there.

I will try the TAR for Windows utility and I have adjusted the values in php.ini to these which should be on the safe side:

max_execution_time = 400

memory_limit = 512M 

The exec() function should be enabled in my php.ini as far as I can see, there are no "disable _functions" enabled in the php.ini so it should be okay. I will try it and see if it works.

Thanks for the great support, creative stuff that really can make a difference! Let's take the next step…Smile

It's late, but Ill post back tomorrow with the results. Thanks again!

 

BTW, here are the details regarding my web setup:

Web Server: IIS 7.5 with FastCGI installed
PHP version: 5.2.11
Joomla: 1.5.22
MySQL: 5.1.1.40 

 

//Marcus

January 13, 2011
9:56 am
Avatar
Ovidiu Liuta
Admin
Forum Posts: 2484
Member Since:
September 26, 2010
sp_UserOfflineSmall Offline

Just to follow up, from what i have been reading, php actually has a limitation to 2GB(unable to create files larger than this) on 32bit systems, is this your case? If so, you might need to upgrade to a 64bit OS along with the appropriate php install to get that fixed.

The only solution for this(without reinstalling the 64bit version) would be to use TAR server utilities i pointed out above, so XCloner uses that external utility to write files and not php internal functions.

 

Ovidiu

January 13, 2011
5:42 pm
Avatar
Marcus
Member
Members
Forum Posts: 20
Member Since:
January 12, 2011
sp_UserOfflineSmall Offline

I also considered PHP to be a limiting factor, but my system and OS are 64-bit all the way. Cramming 16 GB of RAM into a 32-bit system could be difficult.. Wink

I'm just about to start downloading and test the TAR server utility and see how that goes, I'll report back here soon with the results.

I can almost see the light at the end of the tunnel Laugh

//Marcus 

January 13, 2011
6:31 pm
Avatar
Marcus
Member
Members
Forum Posts: 20
Member Since:
January 12, 2011
sp_UserOfflineSmall Offline

Here's what I have done so far:

  1. Downloaded TAR from the link you provided, I chose the Setup Binaries (tar-1.13-1-bin.exe)
  2. Installed TAR on the web server using default values.
  3. After installation, checked the file path to tar.exe and it was correctly "C:\program files (x86)\gnuwin32\bin\tar.exe"
  4. Configured the settings in Xcloner by checking the box "Active" in the section "Tar path or command" and entering the file path like this: C:/program files (x86)/gnuwin32/bin/tar.exe and then pressed "Save". Configuration successfully updated.
  5. I the generated a backup, I click "continue" on the first page but get an error on the next page that says Backup failed, please check your tar server utility support! 
  6. I go back and double check the config and started some light troubleshooting (see below)

Okay, so I tried a few things for troubleshooting:

  • The first thing I suspected to cause a problem were the spaces in the TAR file path (C:/program files...) so I put the file path between "" signs like "C:/program files (x86)/gnuwin32/bin/tar.exe", still problems.
  • I uninstalled TAR and reinstalled it in another path without spaces, "C:/gnuwin32". Changed the config in Xcloner to reflect the new path, but with the same error. I also tried removing the trailing ".exe" in the file path.
  • I tried using backslashes in the file path ("C:\gnuwin32\bin\tar.exe"). I attempted to find an example of how to write the file path and command in the Xcloner User Guide but saw only the instructions to add a path or command, not how it could be done.

So after a few searches on Google, here I am again Laugh Have I got it all wrong?

//Marcus

January 13, 2011
6:55 pm
Avatar
Ovidiu Liuta
Admin
Forum Posts: 2484
Member Since:
September 26, 2010
sp_UserOfflineSmall Offline

Well you need to check that the exec() function is working correctly on your php install, check here http://php.net/manua.....exec.php , there are a few examples for IIS, as well as how to add the paths, this might help C:\gnuwin32\bin\tar.exe

I would be able to help better if i would have direct access to the server ... 

If it still fails, your best bet would be to get the backups to fit under 2GB, so excluding 1-2 folders might help

 

Ovidiu

January 13, 2011
7:50 pm
Avatar
Marcus
Member
Members
Forum Posts: 20
Member Since:
January 12, 2011
sp_UserOfflineSmall Offline

I think I'm going off into the deep end here with the PHP stuff. I think I'll stop here. I might do some more troubleshooting, but for now I'll settle with excluding folders to keep the size down. It's not optimal but doable.

Ovidiu, you should have all the cred in the world for your excellent free support! I have made a small Paypal donation as a token of gratitude. I'll also make sure to give many thumbs up over at JED where I found Xcloner.

For now, thank you for the support!

//Marcus

Forum Timezone: America/Chicago
Most Users Ever Online: 867
Currently Online:
Guest(s) 1
Currently Browsing this Page:
1 Guest(s)
Top Posters:
mlguru: 30
Django29: 29
Andy: 21
D: 21
Marcus: 20
Jamie F: 19
Member Stats:
Guest Posters: 738
Members: 10030
Moderators: 2
Admins: 3
Forum Stats:
Groups: 3
Forums: 7
Topics: 2397
Posts: 8236
Newest Members:
Omar Vera
Moderators: TriP: 0, Steve Burge: 0
Administrators: Ovidiu Liuta: 2484, Victor Drover: 1, Valentin Barbu: 0