acefael

.. on software

Follow me on GitHub

ds_file_iterator

24 Mar 2015

The day comes when you want to backup an entire VMDS SOM or take a snapshot copy to test your upgrade. How would you do that? Shut everything down and use Explorer? Backup Manager? Iterate files on the filesystem and ds_transfer stuff out? What about Superfiles?

Most of these things are live and working somewhere. They are all fine in their place. One small thing that has been bugging me for a while is how to correctly copy out Superfiles, because you cannot reliably tell their size limit by looking at the filesystem alone.

The API

As it so happens, Smallworld™ GIS has an API to go over the particulars of datastore files.

It boils down to SOC → SOM → ds_file → Superfile components. The Superfile components ‘know’ their size limit.

With that it is possible to loop over known datasets and extract the information from there:

  _for ds _over gis_program_manager.cached_datasets()
  _loop
    _if ds.responds_to?( :files )
    _then
      _for f _over ds.files.elements()
      _loop
        _local (path_name,regular?,convertable?,readonly?,temporary?) <<
          f.superfile_query_file()

          ...

          _local component_info << _unset

          _if regular? _isnt _true
          _then
            _local ( file_id, block_id, comp_total, comp_max ) <<
              f.superfile_query_index()

            ...

            component_info << sw:vector.new( comp_total )

            _for i _over range(1, comp_total)
            _loop
              _local ( pname , max_size , used_size ) <<
                f.superfile_query_component( i-1 )
              component_info[ i ] << { pname , max_size , used_size }
            _endloop
          _endif

          # do *something* to the file

        _endif
      _endloop
    _endif
  _endloop

The critical pieces are those .superfile_query_*() methods on ds_file. Notheworthy is the check for support of the .files protocol on the SOM. This serves to establish we are looking at a VMDS / SWDP SOM.

How To Use

In above code, where it reads “do *something* to the file”, the things available to make use of are f (a ds_file) and components (a simple_vector, or _unset). ds_file_iterator.magik contains the full implementation of the iterator method ds_file_iterator.each and a tried and tested way to use it.

ds_file_iterator.each takes as argument a _proc that is being called with exactly a ds_file and the components of that superfile, if any.

The sample method ds_file_iterator.backup_file shows how to apply .each. If you set $DATABASE_ROOT and $BACKUP_ROOT accordingly it can copy your files from c:\Smallworld\cambridge_db\ds\ds_gis\rwo.ds to c:\Smallworld\backup\ds\ds_gis\rwo.ds. It uses ds_transfer for that.

The rest is details. It’s in above mentioned file. Important is: you won’t ever have to worry about added Superfile components anymore.

How to Improve

ds_file_iterator.each can obviously only iterate VMDS SOMS. Anything else is simply skipped. It is blissfully unaware of files like replication_metadata.ds.

This could have been an _iterator _method.

The Long End

This has been tested on 4.11.

Be sure to let me know if you like it!