When tracking third party software in Fossil, I want to have better support for checking external source in in full.

Handling it manually is tiresome:
<pre>
fossil open vendor --keep
fossil add $(fossil extras --dotfiles)
fossil rm $(fossil changes | awk '$1 == "MISSING" {print $2}'
fossil ci -m "Import vendor version 6.6.6"
</pre>
The shell magic is error-prone, it works badly with files containing white space in names, it is good luck I haven't run into problems yet.

I propose to create some "import" command that synchronizes inventory with file system.

I support this. The import command should also add a non-propagating tag like CVS does to make it easier to keep that of what version was imported.

I have added an experimental IMPORT command to my branch wolfgangHelpCmd (checkin [b37d5f256e]).

Maybe you take a look at it and/or test it in your environment.

The command help is:

<pre>
Usage: fossil.exe import

If used in a checkout, the current checkout file tree is synchronized
with the repository:
 * all files, existing in the file tree but not in the repository
   (files displayed using the "extra" command)
   are added to the repository - see also "add"
 * all files, existing in the repository, not existing in the file tree
   (files displayed as MISSING using the "status" command)
   are removed from the repository - see also "rm"
The command does not "commit"!

This command can be used to track third party software.
</pre>

best regards


Well, coming from the cvs model, effective 3rd party management includes:

(1) A separate branch with the unmodified sources in a specific location.
Ideally, the import command does a sanity check that the "vendor" branch doesn't contain anything else.

(2) A symbol for each each imported version. Normally, this is the release version from upstream or the commit id or whatever. This should be a non-propagating tag.

(3) A merge from the vendor branch to trunk. As fossil keeps track of the merge state, this is easier than with cvs, where it had to be simulated with cvs up -j old-tag -j new-tag.

A really useful "fossil import" would therefore be:

    fossil import REPOSITORY PATH BRANCH TAG

and not require a checkout of the proper branch. The merge part is easy enough to do with the normal merge processing and can be skipped.

Separate branch is easy to create yourself, the problem is that you can't merge from it even to empty trunk:
<pre>
$ fossil merge vendor
fossil: lack both primary and secondary files
</pre>
Perhaps this should be fixed too.

A symbol for each version isn't needed, you have version identifiers already,
and you can always tag what you want.


I think - like mentioned above - creating a branch and tag with proper vendaor tags can already be done using the standard fossil commands.

The main problem, deescribed at the beginning was the sync of the checkout file tree and the files in the repository. This can be hanlded with the new commen.

Would the merge work, if we use the cherrypicck option? I cannot test it now, but i'll try it later and i think, it will work.

The only problem, we may have is the fact, that the import command needs to be called from an ceckout. The new command shares most of its code withalready exitant code for and all these ccommands are only implemented, when called from an existing checkout.
It will be complex to extendd it to work with standalone checkouts, and i don't believe, that it's neccesary:

  *  First checkin:<br>fossil new x.fsl;mkdir x;cd x;fossil open ..\x.fsl<br>
     copy source tree to x<br>fossil import<br>fossil commit -m "..."<br>and
     set branches/tagss to whatever you want
  *  further checkins:<br>cd x;fossil update VENDORBRANCH<br>
     remove old data and copy new source tree<br>
     fossil import;fossil commit -m ".."

Merging to main development branch should be possible by using cherrypick

best regards


Re tagging the import:

I've had to deal with vendor imports that often enough lack any version number other than some RCS ids in the file. Importing files directly from the upstream version control system is a common task for larger projects. Sadly enough, having to identify what changed between different imports is also often enough required.

Re main problem:

I consider "import" a convenience command. The original set of commands can be simplified a lot by removing all files first with find . -type f -print0 | xargs -0 fossil rm, copying the new files and doing find . -type f -print0 | xargs -0 fossil add. That's safe for spaces in names etc. Having to switch to the vendor branch requires either creating another checkout and copying files or switching from a large checkout to a minimal one. The latter isn't a lot of fun if you have a working copy size of > 100MB and uncommitted changes.

Not having to move the vendor branch has another very property, it reduces the chance of accidentally messing it up.

Re fossil merge failure:

This looks like a bug. Merging to the initial empty version should certainly be allowed as special case. It works fine if any commit was done on trunk first.

Following my proposol, you could see any changes between different imports. Though we add/remove only some of the file, the commit will also track the differences between changed files. Creating a vendor branch on the first import and using a tag like VENDOR_1 and using new tags on this branch for following import (like VENDOR_2). All changes (new/rm and change) can be seen using "fossil diff --from VENDOR_1 --to VENDOR_2". As long, as we use the vendor branch only for the checkin's, we can also see them in the gui.

The import command is useful on OS != *nix. On windows, scripting these commands is a little bit tricky. Because add/extras can work recursiv, i think it's a good extension, to offer the mixed comman import.

I would always use multiple checkouts, to handle venddor imports - on the vendor branch - and development - main trunk or development branch.

Merge bug filed separatly as
Test case provided.

If you just want to synchronize the file list with the working copy, a better command would be to adopt Mercurial's addremove.

I maintain that having to use a separate working copy for doing vendor imports is a PITA.

The new import program works like "mercurial addremove", so i think, i should rename the command.

I've started a small test repository on
  *
  *  use user/password d/d to play with it.

import/addremove and merging with cherrypick seems to work fine for me.

But i can see some problems/weak points:
  *  importing a new delivery needs to remove unused file from the source tree.
     For big projects, this might only be done by deleting the complete tree
  *  if the development branch is messed up with temporary files, which are
     not under version control, it's difficult to remove them

This could only be done, using a import directory outside of the checkout.

Merging is OK for me.

Conclusion:
  *  I'll rename import to addremove
  *  I'll analyse the code, how to do a add/remove from outside a repository.
     But this will be difficult, because i cannot use existing code. But i
     think, it will be easier to add an option --keep to update(like for open).
     So we could<br>
     1. create a new directory for the delivery<br>
     2. go to this directory and open a fossil connection with --keep<br>
     3. update this <i>checkout</i> to VENDOR without affecting the file using
        --keep<br>
     4. commit and tag the new checkin with VENDOR_n+1