CRUX

Welcome to CRUX bug tracking.
Tasklist

FS#1391 - faster rsync driver

Attached to Project: CRUX
Opened by Fun (fun) - Tuesday, 07 March 2017, 00:39 GMT
Last edited by Fredrik Rinnestam (frinnst) - Saturday, 17 June 2017, 16:49 GMT
Task Type Improvement
Category ports → core/opt
Status Closed
Assigned To Juergen Daubert (jue)
Operating System CRUX
Severity Low
Priority Normal
Reported Version 3.3
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

I've attached a patch for the rsync driver, to synchronize ports in half the time.

The original driver does:
1) read the list of current files
2) 'download' the list of source files (~0.5MB for core,opt,xorg,compat-32,contrib)
3) synchronize
4) write the new list of files
5) remove the files/dirs not found in the new list

The new driver:
- removes (2); after synchronization it can build the list from the local files
- optimize (4); use a single write to disk
- optimize (5); it doesn't need to sort the list (maybe it doesn't worth this much)
   rsync.patch (1.8 KiB)
This task depends upon

Closed by  Fredrik Rinnestam (frinnst)
Saturday, 17 June 2017, 16:49 GMT
Reason for closing:  Won't implement
Comment by Fredrik Rinnestam (frinnst) - Saturday, 11 March 2017, 00:43 GMT
Seems to be working fine for me
Comment by Fredrik Rinnestam (frinnst) - Monday, 22 May 2017, 07:41 GMT
There seems to be some issues with deleted ports not being actually deleted. See the IRC log for 2017-05-22
Comment by Fun (fun) - Monday, 22 May 2017, 08:07 GMT
Besides rolling back the last patch, here is a completion :)

   0001-Use-delete-when-syncing-... (2 KiB)
Comment by Juergen Daubert (jue) - Monday, 22 May 2017, 10:31 GMT
Please attach a patch with only the fix for the bug found by Fredrick, without adding new "features" that might introduce new bugs. If that's not possible I'll revert your initial patch.

Thanks
Comment by Fun (fun) - Monday, 22 May 2017, 11:24 GMT
Sorry for all this.

About the second patch:
- To fix the current situation (the old files/dirs not being deleted) adding "--delete" to rsync is enough.
- The "--exclude=.checkouts" option was to avoid annoying all the people about being deleted with every update (because it is recreated).
- The "del." to "Delete" replacement on output is just cosmetic, in line with the old way of showing removals.
- The last chunk removed by this patch was dead code. It removes files/dirs that was not removed. After the first patch no file will be seen as need to be removed, and with "--delete" it won't.

About the first patch:
The original driver "downloaded" the original file-list from the server, in order to "manually" remove files/dirs and not let rsync do that.
The first patch "optimized" the process by avoiding the "file-list download", but was incomplete without adding --delete to rsync.
The other two optimizations does much less than this.

Still, I've attached a stripped down patch.
Comment by Fun (fun) - Monday, 22 May 2017, 11:25 GMT
Now...
   0001-Remove-the-old-files-dir... (0.9 KiB)
Comment by Fun (fun) - Monday, 22 May 2017, 13:01 GMT
However, using rsync with --delete will bring the rsync driver in the same place git driver is. See #1313
Comment by Juergen Daubert (jue) - Monday, 22 May 2017, 13:49 GMT
Ok, thanks for the explanation. However, I don't like the idea that we buy speed improvements on cost of a security feature. Removing files/directories within a chroot sounds like a good idea to me.
Comment by Fun (fun) - Monday, 22 May 2017, 14:03 GMT
Just to be clear, the --delete should also remove files only from the specified collection. But, the original version skipped the "alien" files (sources, packages) that might be in that tree. I wasn't really sure you want to keep this, as I didn't see any movement with the git driver in this direction.

My first thought was that you'll do a rollback, but I've completed the previous patch because I've felt I'm responsible.
Comment by Juergen Daubert (jue) - Monday, 22 May 2017, 14:23 GMT
Well, removing sources/packages from the tree is a bad idea as long our default is to use the ports tree for them. Not everybody uses a special directory for sources and/or packages.
Comment by Fredrik Rinnestam (frinnst) - Monday, 22 May 2017, 14:39 GMT
BTW, the git driver leaves a lot to be desired last time I used it. I don't think using that as a reference is a good idea. Won't leaving source and package-files in the tree break the git driver as it is now? Last time i played with it it didnt handle local changes at all.

Loading...