Thursday, September 15, 2016

Setting up ctags - automatically for multiple independent projects

Howdy everyone.

For those of you using vim, you'll know that <C-]> will jump to a function definition.
The problem with this is that to jump between files searching for a definition, ctags files must be hanging around somewhere for vim to find them. There are plenty of guides out there to set it up, including the great guide at http://ctags.sourceforge.net/faq.html.

Unfortunately, this generates a huge tags file at the root of your projects directory saving references to all of your projects. If the projects are independent this is less than desirable.

This post describes how to set up separate tag trees for all of your projects independently.
Note that this assumes that each project is controlled by its own git repository. This method should be easily extendable to any project structure which has a predictable file/directory at its root and nowhere else in the project (if it's git controlled the .git directory fits this nicely).

We will accomplish this automation through 3 steps

  1. Build a script to generate hidden .tag files
  2. Add instructions to our .vimrc to update tags upon file save
  3. Automatically regenerate the full tag tree via cronjobs
1. Scripting the tag generation
-------------------------------------

The script can be found in the attached file updateTags.sh.

The important bits are the following:

TOP_DIR=$1
for rootdir in $(find "$TOP_DIR" -type d -name '.git'); do
  rootdir_nongit=$(echo $rootdir | sed -e 's|/[^/]*$||');
  for subdir in $(find $rootdir_nongit -type d \( -name '.*'                \
                        -o -name 'ext' -o -name 'doc' -o -name 'docs'       \
                        -o -name 'bin' \) -prune -o -type d -print); do
    cd "$subdir"
    ctags -f .tags --format=2 --excmd=mixed --extra=+q+f --fields=nKsaSmtl *;
  done
  cd $rootdir_nongit
  ctags -f .tags --format=2 --excmd=mixed --extra=+q+f --fields=nKsaSmtl    \
        --file-scope=no -R *;
done

The TOP_DIR variable merely is a convenience variable to represent the input, which should
be a directory. We then have the outermost loop, which cycles through all directories within $TOP_DIR which have a .git directory, and saves extracts the names of those variables to the rootdir_nongit variable.
The inner loop cycles through all subdirectories of $rootdir_nongit, and drops ctags files into them with the name .tags (because making them hidden prevents them from cluttering the file list for ls).
After all the subdirectories (except for hidden, ext, doc, docs and bin subdirectories) have ctags files created, a giant ctags file with references for the entire project is dropped in at the $rootdir_nongit level (this is the -R option).
You may want to modify the ctags options, see the man page for a complete description.

The attached file also contains a bunch of logging statements, and clears out any empty ctags files.

2. Telling VIM where to find the tag files
--------------------------------------------------
The following code should be added to your .vimrc file.

i) Set tags to the .tags file in the directory of the file you're editing
let &tags=expand('%:p:h')."/.tags"

ii) Find the .tags file in the root directory when entering a file
function! Setup_tags ()
  for rootDirs in finddir(".git", expand('%:p:h').";~/Projects",-1)
    if strlen(findfile('.tags',rootDirs[0:-5]))
      let &tags .= rootDirs[0:-5] . ".tags,"
    endif
  endfor
endfunction
autocmd VimEnter *.* :call Setup_tags()
This function finds the root directory (containing the .git directory, terminating the search for the root directory at the Projects directory), and adds it's .tags file to the list of tags locations to search. The final command outside of the function actually sets up the tags when vim is entered.

iii) Automatically update the tag file upon save
Since ctags is pretty quick, regenerating the tag file is quick and imperceptible, so I just like to regenerate the local .tags file upon file save.
To ensure that tag files are only updated (not generated if they are missing), add the following:
function! Update_tags ()
  if strlen(findfile('.tags', expand('%:p:h')))
    :silent !(cd %:p:h;ctags -f .tags --format=2 --excmd=mixed --extra=+q+f --fields=nKsaSmtl *)&
  endif
endfunction
autocmd BufWritePost * :call Update_tags()
the silent causes this to happen without prompting the user to acknowledge. The cd %:p:h ensures that the .tags file in the directory of the file you are editing gets regenerated - rather than the directory you entered vim from.

NOTE: If the files you work on are extra large, it may be better to just append the changes to the .tags file (which does not remove tags to functions that were deleted - hence the default of just regenerating the whole thing). To append, add the -a flag to the ctags call.

3. Automatically regenerate the .tags files
--------------------------------------------------
Automatically running scripts is trivial with the magic of cron. Since we have already created an updateTags.sh script, add the following line to your crontab (via crontab -e):
*/30 * * * * ~/updateTags.sh ~/Projects > ~/.local/logs/updateTags.log 2>&1
This line causes the tags to regenerate every half hour (change the 30 to 15 for quarter hour updates, etc). Furthermore, if you are using the linked file with all the logging, all the logged output will be dumped into your ~/.local/logs/updateTags.log file (make sure that the ~/.local/logs file exists before the cron job executes).

Setting up ctags - automatically for multiple independent projects

Howdy everyone.

For those of you using vim, you'll know that <C-]> will jump to a function definition.
The problem with this is that to jump between files searching for a definition, ctags files must be hanging around somewhere for vim to find them. There are plenty of guides out there to set it up, including the great guide at http://ctags.sourceforge.net/faq.html.

Unfortunately, this generates a huge tags file at the root of your projects directory saving references to all of your projects. If the projects are independent this is less than desirable.

This post describes how to set up separate tag trees for all of your projects independently.
Note that this assumes that each project is controlled by its own git repository. This method should be easily extendable to any project structure which has a predictable file/directory at its root and nowhere else in the project (if it's git controlled the .git directory fits this nicely).

We will accomplish this automation through 3 steps

  1. Build a script to generate hidden .tag files
  2. Add instructions to our .vimrc to update tags upon file save
  3. Automatically regenerate the full tag tree via cronjobs
1. Scripting the tag generation
-------------------------------------

The script can be found in the attached file updateTags.sh.

The important bits are the following:

TOP_DIR=$1
for rootdir in $(find "$TOP_DIR" -type d -name '.git'); do
  rootdir_nongit=$(echo $rootdir | sed -e 's|/[^/]*$||');
  for subdir in $(find $rootdir_nongit -type d \( -name '.*'                \
                        -o -name 'ext' -o -name 'doc' -o -name 'docs'       \
                        -o -name 'bin' \) -prune -o -type d -print); do
    cd "$subdir"
    ctags -f .tags --format=2 --excmd=mixed --extra=+q+f --fields=nKsaSmtl *;
  done
  cd $rootdir_nongit
  ctags -f .tags --format=2 --excmd=mixed --extra=+q+f --fields=nKsaSmtl    \
        --file-scope=no -R *;
done

The TOP_DIR variable merely is a convenience variable to represent the input, which should
be a directory. We then have the outermost loop, which cycles through all directories within $TOP_DIR which have a .git directory, and saves extracts the names of those variables to the rootdir_nongit variable.
The inner loop cycles through all subdirectories of $rootdir_nongit, and drops ctags files into them with the name .tags (because making them hidden prevents them from cluttering the file list for ls).
After all the subdirectories (except for hidden, ext, doc, docs and bin subdirectories) have ctags files created, a giant ctags file with references for the entire project is dropped in at the $rootdir_nongit level (this is the -R option).
You may want to modify the ctags options, see the man page for a complete description.

The attached file also contains a bunch of logging statements, and clears out any empty ctags files.

2. Telling VIM where to find the tag files
--------------------------------------------------
The following code should be added to your .vimrc file.

i) Set tags to the .tags file in the directory of the file you're editing
let &tags=expand('%:p:h')."/.tags"

ii) Find the .tags file in the root directory when entering a file
function! Setup_tags ()
  for rootDirs in finddir(".git", expand('%:p:h').";~/Projects",-1)
    if strlen(findfile('.tags',rootDirs[0:-5]))
      let &tags .= rootDirs[0:-5] . ".tags,"
    endif
  endfor
endfunction
autocmd VimEnter *.* :call Setup_tags()
This function finds the root directory (containing the .git directory, terminating the search for the root directory at the Projects directory), and adds it's .tags file to the list of tags locations to search. The final command outside of the function actually sets up the tags when vim is entered.

iii) Automatically update the tag file upon save
Since ctags is pretty quick, regenerating the tag file is quick and imperceptible, so I just like to regenerate the local .tags file upon file save, via the following line:
autocmd BufWritePost * :silent !(cd %:p:h;ctags -f .tags 
                                \--format=2 --excmd=mixed --extra=+q+f 
                                \--fields=nKsaSmtl *)&
the silent causes this to happen without prompting the user to acknowledge. The cd %:p:h ensures that the .tags file in the directory of the file you are editing gets regenerated - rather than the directory you entered vim from.

NOTE: If the files you work on are extra large, it may be better to just append the changes to the .tags file (which does not remove tags to functions that were deleted - hence the default of just regenerating the whole thing). To append, add the -a flag to the ctags call.

3. Automatically regenerate the .tags files
--------------------------------------------------
Automatically running scripts is trivial with the magic of cron. Since we have already created an updateTags.sh script, add the following line to your crontab (via crontab -e):
*/30 * * * * ~/updateTags.sh ~/Projects > ~/.local/logs/updateTags.log 2>&1
This line causes the tags to regenerate every half hour (change the 30 to 15 for quarter hour updates, etc). Furthermore, if you are using the linked file with all the logging, all the logged output will be dumped into your ~/.local/logs/updateTags.log file (make sure that the ~/.local/logs file exists before the cron job executes).

Saturday, May 14, 2016

Python source distribution with fortran dependencies

This post is intended make crystal clear how to distribute python packages which need to interface with a large fortran library. If you haven't already, you should familiarize yourself with the basics of using numpy.distutils with f2py [f2py-distutils documentation].

Are you back now - and fully understood the docs? Good.

Ideally, one can use f2py directives to eliminate the need for a signature file. What follows below will describe how to coax numpy.distutils into building the shared library WITHOUT having to create a signature file.


A Simple Example

Suppose you have the following Code structure (files and modules are 1 to 1 in this example):

src/
 |--pysrc/
 |    |--main.py
 |--forsrc/
 |    |--interface.f08
 |    |--codelogic.f08
 |--setup.py

codelogic.f08 may contain all sorts of derived types which f2py can't really handle. The interface.f08 file should contain subroutines and functions that contain ONLY arrays and default types (real, complex, integer, etc) within their message signatures (inputs and outputs). So we really want f2py to generate a shared library which accesses the interface module, but ignores the codelogic module.

The magic that makes this happen all occurs within the Extension object, inside an option called f2py_opts. Creating the following will do the trick:
    f2py_opts = [':','forsrc/interface.f08']
So within setup.py there will be an Extension object defined as:
    forlib = Extension(name='my_fort_lib',
                       sources=['forsrc/interface.f08',
                                'forsrc/codelogic.f08'],
                       f2py_opts=[':','forsrc/interface.f08'])
Toss this into your setup() function, and you should be able to
  import my_fort_lib
inside of main.py


A Deeper Look

Consider the following Extension:
  forlib = Extension(name='my_fort_lib',
                     sources=['forsrc/interface.f08',
                              'forsrc/codelogic.f08'],
                     extra_f90_compile_args=['-ggdb',-gbacktrace],
                     libraries=['m'],
                     f2py_opts=[':',
                                'forsrc/interface.f08',
                                'skip:',
                                'foo', 'bar'])
Here's a description of each keyword and what options were provided:

  •   name='my_fort_lib' -> The name of the shared library will be such that python can import my_fort_lib
  •   sources=[...] -> the fortran source files
  •   extra_f90_compile_args=[...] -> options passed to the compiler. -ggdb and -gbacktrace enable debugging and backtrace flags for the gcc compilers
  •   libraries -> libraries the linker needs to link against. m is the standard math library
  •   f2py_opts -> the options to pass to f2py
    • ':' -> the following items in the list become the only source files f2py uses when constructing the library. This terminates when another option is found in the list.
    • 'skip:' -> the following items in the list are skipped when constructing the library. These should be subroutine and function names.
    • 'foo', 'bar' -> subroutine names not exposed to the shared library available to python