syconn.utils

syconn.utils.basics

class syconn.utils.basics.Coordinate(c)

Bases: list

Represents a coordinate of arbitrary dimensionality.

class syconn.utils.basics.FloatCoordinate(c)

Bases: syconn.utils.basics.Coordinate

Represent a coordinate of arbitrary dimensionality, using floats.

syconn.utils.basics.average_coordinate(c)

Return the average coordinate (center of gravity) for an iterable of coordinates.

Parameters:c (iterable of coordinates) – Coordinates are represented as lists and must have the same number of dimensions. Returns:avg_coordinate Return type:iterable

Example

>>> average_coordinate([[1, 2, 3], [4, 5, 6]])
[2.5, 3.5, 4.5]
>>> average_coordinate([])
[]

syconn.utils.basics.calc_overlap(point_list_a, point_list_b, max_dist)

Calculates the portion of points in list b being similar (distance max_dist) to points from list a. :param point_list_a: :param point_list_b: :param max_dist: :return: Portion of similar points over number of points of list b and vice versa, overlap area in nm^2, centercoord of overlap area and coord_list of overlap points in point_list_b

syconn.utils.basics.cell_object_coord_parser(voxel_tree)

Extracts unique voxel coords from object tree list for cell objects ‘mitos’, ‘vc’ and ‘sj’.

Parameters:voxel_tree – annotation object containing voxels of cell objects [‘mitos’, ‘vc’, ‘sj’] Returns:coord arrays for ‘mitos’, ‘vc’ and ‘sj’

syconn.utils.basics.convex_hull_area(pts)

Calculates the surface area from a given point cloud using simplices of its convex hull. For the estimation of the synapse contact area, divide by a factor of two, in order to get the area of only one face (we assume that the contact site is sufficiently thin represented by the points). :param pts: np.array of coordinates in nm (scaled) :return: Area of the point cloud (nm^2)

syconn.utils.basics.coordinate_from_string(coord_string)

syconn.utils.basics.coordinate_to_ewkt(coordinate, scale='dataset')

syconn.utils.basics.euclidian_distance(c1, c2)

syconn.utils.basics.get_box_coords(coord_array, min_pos, max_pos, ret_bool_array=False)

Reduce coord_array to coordinates in bounding box defined by global variable min_pos max_pos. :param coord_array: array of coordinates :return:

syconn.utils.basics.get_normals(hull, number_fitnodes=12)

Calculate normals from given hull points using local convex hull fitting. Orientation of normals is found using local center of mass. :param hull: 3D coordinates of points representing cell hull :type hull: np.array :return: normals for each hull point

syconn.utils.basics.get_orth_plane(node_com)

Calculates orthogonal plane and skeleton interpolation for each node. :param node_com: Spatially ordered list of nodes :return: orthogonal vector and vector representing skeleton interpolation at each node

syconn.utils.basics.has_equal_dimensions(c)

Return True if container types in iterable c have equal number of of elements, False otherwise.

Example

>>> a = set(range(0, 10))
>>> b = range(0, 10)
>>> has_equal_dimensions([a, b])
True
>>> a.add(100)
>>> has_equal_dimensions([a, b])
False

syconn.utils.basics.helper_samllest_dist(args)

Returns the smallest distance of index ixs in dists. :param ixs: list of in Indices of objects :param annotation_ids: array of shape (m, ) :param dists: array of shape (m, ) :return: smallest distance of ixs found in dists.

syconn.utils.basics.interpolate_between(c1, c2)

Return list of coordinates from c1 to c2, including them. Coordinates are spaced by (approximately) one length unit.

syconn.utils.basics.negative_to_zero(a)

Sets negative values of array a to zero. :param a: numpy array :return: array a with non negativ values.

syconn.utils.basics.poly_area(poly)

Calculates the area of a given polygon. :param poly: list of points :return: area of input polygon

syconn.utils.basics.pre_process_volume(vol)

Processes raw data to get membrane shapes. :param vol: array raw data :return: array membrane

syconn.utils.basics.rotation_matrix(axis, theta)

Get rotation matrix along axis and angle

Parameters:

• axis (np.array) – rotation-axis
• theta (float) – angle to rotate

Returns:

rotation matrix associated with counterclockwise rotation about the given axis by theta radians.

Return type:

np.array

syconn.utils.basics.switch_array_entries(this_array, entries)

syconn.utils.basics.tuple_to_string(coordinate, sep=', ', dl='(', dr=')')

syconn.utils.basics.unit_normal(a, b, c)

Calculates the unit normal vector of a given polygon defined by the points a,b and c. :param a, b, c: Each is an array of length 3 :return: unit normal vector

syconn.utils.datahandler

class syconn.utils.datahandler.DataHandler(wd, scaling=(9.0, 9.0, 20.0))

Bases: object

Initialized with paths or cell components (SegmentationObjects), path to membrane prediction and source path of traced skeletons (to be computed). DataHandler is needed for further processing.

wd

str – path to working directory, by default all other paths will lead to subfolders of this directory

mem_path

str – path to barrier prediction

cs_path

str – path to supervoxel data

myelin_ds_path

str – path to myelin prediction

skeleton_path

str – path to cell tracings

data_path

str – output directory for mapped cell tracings

scaling

np.array – scaling of dataset, s.t. after applying coordinates are isotropic and in nm

mem

KnossosDataset – will be assigned automatically

mem_offset

optional offeset of dataset

mitos/vc/sj

UltrastructuralDataset – Dataset which contains cell objects of mitochondria, vesicle clouds and synaptic junctions respectively

syconn.utils.datahandler.cell_object_id_parser(obj_trees)

Extracts unique object ids from object tree list for cell objects ‘mitos’, ‘vc’ and ‘sj’

Parameters:obj_trees (list of annotation objects ['mitos', 'vc', 'sj']) – Returns:keys ‘mitos’, ‘vc’ and ‘sj’ containing unique object ID’s as values Return type:dict

syconn.utils.datahandler.connect_soma_tracing(soma)

Connect tracings of soma, s.t. a lot of rays are emitted. Connects nearby

(kd-tree) nodes of soma tracings with edges

Parameters:soma (SkeletonAnnotation) – Soma tracing Returns:Sparsely connected soma tracing Return type:SkeletonAnnotation

syconn.utils.datahandler.cut_array_in_one_dim(array, start, end, dim)

Cuts an array along a dimension

Parameters:

• array (np.array) –
• start (int) –
• end (int) –
• dim (int) –

Returns:

array

Return type:

np.array

syconn.utils.datahandler.get_filepaths_from_dir(directory, ending='k.zip', recursively=False)

Collect all files with certain ending from directory.

Parameters:

• directory (str) – path to lookup directory
• ending (str) – ending of files
• recursively (boolean) – add files from subdirectories

Returns:

paths to files

Return type:

list of str

syconn.utils.datahandler.get_paths_of_skelID(id_list, traced_skel_dir)

Gather paths to kzip of skeletons with ID in id_list

Parameters:

• id_list (list of str) – skeleton ID’s
• traced_skel_dir (str) – directory of mapped skeletons

Returns:

paths of skeletons in id_list

Return type:

list of str

syconn.utils.datahandler.get_skelID_from_path(skel_path)

Parse skeleton id from filename

Parameters:skel_path (str) – path to skeleton Returns:skeleton ID Return type:int

syconn.utils.datahandler.helper_get_hull_voxels(obj)

Helper function to receive object hull voxels. :param obj: :type obj: SegmentationObject

Returns:object hull voxels Return type:np.array

syconn.utils.datahandler.helper_get_voxels(obj)

Helper function to receive object voxel

Parameters:obj (SegmentationObject) – Returns:object voxels Return type:np.array

syconn.utils.datahandler.hull2text(hull_coords, normals, path)

Writes hull coordinates and normals to xyz file. Each line corresponds to coordinates and normal vector of one point x y z n_x n_y n_z.

Parameters:

• hull_coords (np.array) –
• normals (np.array) –
• path (str) –

syconn.utils.datahandler.load_anno_list(fpaths, load_mitos=True, append_obj=True)

Load nml’s given in nml_list and append according hull information from xyz-file, as well as object coordinates from txt files in kzip (if given).

Parameters:

• fpaths (list of str) – paths to tracings
• load_mitos (bool) – load mito hull points and ids
• append_obj (bool) – load mapped cell objects

Returns:

list of mapped cell tracings

Return type:

list of list of SkeletonAnnotations

syconn.utils.datahandler.load_files_from_kzip(path, load_mitos)

Load all available files from annotation kzip

Parameters:

• path (str) – path to kzip
• load_mitos (bool) – load mito hull points and ids

Returns:

coordinates [hull, mitos, vc, sj], id lists, normals

Return type:

list of np.array

syconn.utils.datahandler.load_from_h5py(path, hdf5_names=None, as_dict=False)

Loads data from a h5py File

Parameters:

• path (str) –
• hdf5_names (list of str) – if None, all keys will be loaded
• as_dict (boolean) – if False a list is returned

Returns:

data

Return type:

dict or np.array

syconn.utils.datahandler.load_mapped_skeleton(path, append_obj, load_mitos)

Load mapped cell tracing and append mapped cell objects

Parameters:

• path (str) – path to tracing kzip
• append_obj (bool) – append cell objects
• load_mitos (bool) – load mitochondria objects

Returns:

mapped cell tracings (skel, mito, vc, sj, soma)

Return type:

list of SkeletonAnnotations

syconn.utils.datahandler.load_objpkl_from_kzip(path)

Loads object pickle file from mapped skeleton kzip

Parameters:path (str) – Path to kzip Returns: Return type:list of SegmentationObjectDataset

syconn.utils.datahandler.load_ordered_mapped_skeleton(path)

Load nml of mapped skeleton and order trees

Parameters:path (str) – Path to nml Returns:List of trees with order [skeleton, mitos, vc, sj, soma] Return type:list

syconn.utils.datahandler.load_pkl2obj(path)

Loads pickle file of object

Parameters:path (str) – path of source file Returns: Return type:UltrastructuralDatasetObject

syconn.utils.datahandler.obj_hull2text(id_list, hull_coords_list, path)

Writes object hull coordinates and corresponding object ids to txt file. Each line corresponds to id and coordinate vector of one object: id x y z

Parameters:

• id_list (np.array) –
• hull_coords_list (np.array) –
• path (str) –

syconn.utils.datahandler.remove_from_zip(zipfname, *filenames)

Removes filenames from zipfile

Parameters:

• zipfname (str) – Path to zipfile
• filenames (list of str) – files to delete

syconn.utils.datahandler.save_to_h5py(data, path, hdf5_names=None)

Saves data to h5py File

Parameters:

• data (list of np.arrays) –
• path (str) –
• hdf5_names (list of str) – has to be the same length as data

Returns:

Return type:

nothing

syconn.utils.datahandler.supp_fname_from_fpath(fpath)

Returns supported filename from path to written file to write it to kzip

Parameters:fpath (str) – path to file Returns:filename if supported, else ValueError Return type:str

syconn.utils.datahandler.switch_array_entries(this_array, entries)

Switches to array entries

Parameters:

• this_array (np.array) –
• entries (list of int) –

Returns:

this_array

Return type:

np.array

syconn.utils.datahandler.write_data2kzip(kzip_path, fpath, force_overwrite=False)

Write supported files to kzip of mapped annotation

Parameters:

• kzip_path (str) –
• fpath (str) –
• force_overwrite (bool) –

syconn.utils.datahandler.write_obj2pkl(objects, path)

Writes object to pickle file

Parameters:

• objects (UltrastructuralDatasetObject) –
• path (str) – destianation

syconn.utils.segmentationdataset

class syconn.utils.segmentationdataset.SegmentationObject(obj_id, path_dataset, path_to_voxels)

Bases: object

add_path(other_obj)

add_voxel(coordinate)

bounding_box

calculate_rep_coord(calculate_size=False, voxels=None, sample_size=200)

calculate_size(voxels=None)

check_if_coordinate_is_contained(coordinate)

closest_point(coordinate, scaling=None)

create_hull_ids()

create_hull_voxels()

hull_voxels

most_distant_voxel

obj_id

path_dataset

path_to_hull_voxels

path_to_voxel

remove_voxel_file()

rep_coord

scaling

size

voxels

write_to_cube(kd, mags=None, size_filter=0, entry=None, as_raw=False, overwrite=True)

write_to_overlaycube(kd, mags=None, size_filter=0, entry=None)

class syconn.utils.segmentationdataset.UltrastructuralDataset(obj_type, rel_path_home, path_to_chunk_dataset_head)

Bases: object

chunk_dataset

find_object_with_coordinate(coordinate, initial_radius_nm=1000, hull_search_radius_nm=500, hull_reduction_by=20, scaling=None, return_if_contained=True)

ids

init_properties()

knossos_dataset

path

rep_coords

sizes

type

syconn.utils.segmentationdataset.check_all_hulls(object_dataset_path, nb_processes, use_qsub=False, queue='full')

syconn.utils.segmentationdataset.check_all_hulls_loop(object_dataset_paths, nb_processes=1, use_qsub=False, queue='full')

syconn.utils.segmentationdataset.check_all_hulls_thread(args)

syconn.utils.segmentationdataset.extract_and_save_all_hull_voxels(object_dataset_path, overwrite=False, nb_processes=1, use_qsub=False, queue='full')

syconn.utils.segmentationdataset.extract_and_save_all_hull_voxels_loop(object_dataset_paths, overwrite=False, nb_processes=1, use_qsub=False, queue='full')

syconn.utils.segmentationdataset.extract_and_save_all_hull_voxels_thread(args)

syconn.utils.segmentationdataset.get_rel_path(obj_name, filename, suffix='')

Returns path from ChunkDataset foler to UltrastructuralDataset folder

Parameters:

• obj_name (str) – ie. hdf5name
• filename (str) – Filename of the prediction in the chunkdataset
• suffix (str) – suffix of name

Returns:

rel_path – relative path from ChunkDataset folder to UltrastructuralDataset folder

Return type:

str

syconn.utils.segmentationdataset.load_dataset(path, use_shelve=False)

syconn.utils.segmentationdataset.path_in_voxel_folder(obj_id, chunk_nb)

syconn.utils.segmentationdataset.save_dataset(object_dataset, path='auto')

syconn.utils.segmentationdataset.update_dataset(object_dataset, update_objects=False, recalculate_rep_coords=False, overwrite=False)

syconn.utils.segmentationdataset.update_multiple_datasets(paths, update_objects=False, recalculate_rep_coords=False, nb_processes=1, use_qsub=True, queue='full')

syconn.utils.segmentationdataset.updating_segmentationDatasets_thread(args)

syconn.utils.skeleton

class syconn.utils.skeleton.Skeleton

Basic class for cell tracings used in Knossos

annotations

set of SkeletonAnnotations – several tracings of one cell

scaling

tuple – scaling of dataset

add_annotation(annotation)

Add SkeletonAnnotation annotation to the skeleton. Node IDs in annotation may not be preserved.

fromDom(doc, use_file_scaling=False, scaling='dataset', comment=None, meta_info_only=False)

fromNml(filename, use_file_scaling=False, scaling='dataset', comment=None, meta_info_only=False)

fromNmlString(nmlString, use_file_scaling=False, scaling='dataset', comment=None, meta_info_only=False)

fromNmlcTree(filename, use_file_scaling=False, scaling='dataset', comment=None)

Reads nml file with cElementTree Parser is capable of parsing patches

Parameters:

• filename (str) – path to nml-file or path to k.zip-file (but this might cause trouble with other functions since self.filename = *.k.zip)
• use_file_scaling (bool) –
• scaling (str) –
• comment (str) –

fromSWC(path='')

SWC is a standard skeleton format, very similar to nml. :param path: str :return:

getAnnotations()

getIdleTime()

getNodes()

getSkeletonTime()

getVolumes()

get_high_node_id()

Return highest node ID in any annotation in the skeleton.

get_version()

has_node(node)

reset_all_ids()

Rebases all node IDs, starting at 1 again. We should have just one node ID and one annotation ID instead of this mess with base and high IDs and only make the node IDs knossos-compatible when writing to nmls. annotation.high_id : highest node id in this annotation annotation.nodeBaseID : lowest node id in this annotation

reset_all_times()

set_edit_position(edit_position)

set_scaling(scaling)

toNml(filename, save_empty=True)

toSWC(path='')

SWC is a standard skeleton format, very similar to nml, however, it is edge-centric instead of node-centric. Some spec can be found here: http://research.mssm.edu/cnic/swc.html SWC can also be read by Amira :param path: str :return:

to_kzip(filename, save_empty=True, force_overwrite=False)

Similar to self.toNML, but writes NewSkeleton to k_zip. :param filename: Path to k.zip :param save_empty: like in self.to_xml_string :param force_overwrite: overwrite existing .k.zip :return:

to_xml_string(save_empty=True)

class syconn.utils.skeleton.SkeletonAnnotation

addEdge(node, target_node)

addNode(node)

add_path(path)

Add all nodes in path to the annotation, connecting them linearly.

Parameters:path (List of SkeletonNode objects) – Nodes that will be added to annotation, connected in order.

appendComment(comment)

avg_inter_node_distance(outlier_distance=2000.0)

Calculates the average inter node distance for an annotation. Candidate for inclusion into the skeleton annotation object.

Parameters:

• anno – annotation object
• outlier_filter – float, ignore distances higher than value

Returns:

float

clearEdges()

clearNodeEdges(node)

clear_nodes()

fromNml(annotation_elem, skeleton, base_id=0)

fromNmlcTree(annotation_elem, skeleton, base_id=0)

Subfunction of fromNmlcTree from NewSkeleton

Parameters:

• annotation_elem (XML Element) –
• skeleton (Skeleton) –
• base_id (int) –

getComment()

getEdges()

getNodeBaseID()

getNodeByID(nodeID)

getNodeByUniqueID(uniqueNodeID)

getNodeEdges(node)

getNodeReverseEdges(node)

getNodes()

getReverseEdges()

getRoot()

getVolumes()

has_node(node)

interpolate_nodes(max_node_dist_scaled=50)

isEmpty()

iter_edges()

physical_length()

Return the physical length along all edges in the annotation.

removeEdge(node, target_node)

removeNode(node)

resetObject()

resetRoot(new_root)

setComment(comment)

setNodeBaseID(nodeBaseID)

setRoot(root)

setRootInternal(root)

toNml(doc, annotations_elem, comments_elem, annotation_ID)

unRoot()

unRootInternal()

class syconn.utils.skeleton.SkeletonLoop

Class for handling loops of patches

add(point)

calculateReducedPoints()

Creates new pointset without double appearing points

fillHole(curr, prior)

Adds a point to a hole detected by fromNmlcTree()

The added point has to be a neighbour of curr (= current point) to guarantee a successful use of the while loop.

fromNmlcTree(loop_elem)

Subfunction of fromNmlcTree from SkeletonVolume

Parameters:loop_elem (XML Element) –

getLimits()

getPoints()

getReducedPoints()

getVP()

resetObject()

setLimits(point)

Calculates the two outer points (3D) of the box exactly around the loop

setVP(vp)

class syconn.utils.skeleton.SkeletonLoopPoint

Class for handling points of loops

fromNmlcTree(point_elem)

Subfunction of fromNmlcTree from SkeletonLoop

Parameters:point_elem (XML Element) –

getCoordinates()

resetObject()

setCoordinates(point_elem)

Sets the coordinates for a point and convert them to int

class syconn.utils.skeleton.SkeletonNode

addChild(child)

addMetaDataElem(md_elem)

addParent(parent)

appendComment(comment)

degree()

delCommentPart(comment)

detachAnnotation()

distance_scaled(to_node)

fromNml(annotation, node_elem)

fromNmlcTree(annotation, node_elem)

from_coordinate(coordinate)

from_scratch(annotation, x, y, z, inVp=1, inMag=1, time=0, ID=None, radius=1.0)

getChildren()

getComment()

getCoordinate()

getCoordinate_scaled()

getData()

getDataElem(elem_name)

getID()

getMetaData()

getNeighbors()

getParents()

getPureComment()

getSingleParent()

getUniqueID()

has_comment_token(token)

implyMetadataAddRoot()

implyMetadataRemoveRoot()

removeChild(child)

removeMetaData()

removeMetaDataKey(md_key)

removeParent(parent)

removeSingleParent()

resetObject()

setComment(comment)

setCoordinate(coord)

setData(data)

setDataElem(elem_name, elem_value)

setID(id)

setMetaData(md)

setPureComment(comment)

setRoot()

setSingleParent(parent)

toNml(doc, nodes_elem, edges_elem, comments_elem)

unRoot()

class syconn.utils.skeleton.SkeletonVolume

Class for handling patches (=volumes)

fromNmlcTree(patch_elem)

Subfunction of fromNmlcTree from SkeletonAnnotation

Parameters:patch_elem (XML Element) –

getComment()

getLimits()

getLoops()

getVP()

resetObject()

setComment(comment)

setLimits()

Calculates the two outer points (3D) of the box exactly around the volume

setVP(vp)

class syconn.utils.skeleton.Version

Bases: list

syconn.utils.skeleton.build_attributes(xml_elem, attributes)

syconn.utils.skeleton.compare_version(v1, v2)

Return ‘>’, ‘<’ or ‘==’, depending on whether the version represented by the iterable v1 is larger, smaller or equal to the version represented by the iterable v2.

Versions are represented as iterables of integers, e.g. (3, 4, 2).

syconn.utils.skeleton.file_md5_checksum(f)

“Computes the MD5 hash for a file

syconn.utils.skeleton.from_id_lists_to_mergelist(id_lists, coordinates, path, immutable=1)

syconn.utils.skeleton.from_skeleton_to_mergelist(cset, skeleton_path, name, hdf5_name, obj_coordinates=None, objs=None, mergelist_path=None, nb_processes=1)

syconn.utils.skeleton.id_lookup_in_one_chunk_thread(args)

syconn.utils.skeleton.integer_checksum(i)

Computes the SHA256 hash of an integer.

syconn.utils.skeleton.parse_attributes(xml_elem, parse_input)

syconn.utils.skeleton.parse_cET(xml_elem, parse_input, ret_all_attr=False)

Implementation of cElementTree Parser

Keyword arguments: xml_elem – XML Element parse_input – what the parser should look for [“whatyouarelookingfor”, type]

returns a list of python-typed values

syconn.utils.skeleton.remove_from_zip(zipfname, *filenames)

Removes filenames from zipfile. :param zipfname: str Path to zipfile :param filenames: list of str Files to delete

syconn.utils.skeleton_utils

class syconn.utils.skeleton_utils.AnnotationSearch(annotation, node, f, context)

Bases: object

BFS(descend_f)

annnotationChildrenBFS()

annnotationNeighborsBFS()

annnotationParentsBFS()

bfs_search(from_node, to_visit, next_nodes)

dfs_search(from_node, to_visit, next_nodes)

search(search_f, next_f, process_f)

class syconn.utils.skeleton_utils.EnhancedAnnotation(annotation, interpolate=False)

Representation of an annotation that additionally includes a NetworkX graph representation and KD-Tree (from scipy spatial module) representation of the same annotation.

get_shortest_path(from_node, to_node)

Return EnhancedAnnotation object representing the segment defined by the shortest path between from_node and to_node.

Parameters:

• from_node (SkeletonNode object) – Defines the beginning of the path
• to_node (SkeletonNode object) – Defines the end of the path

exception syconn.utils.skeleton_utils.InvalidFileFormatException

Bases: exceptions.Exception

class syconn.utils.skeleton_utils.KDtree(nodes, coords=None)

scipy based KD-tree wrapper class that allows efficient spatial searches for arbitrary python objects.

queryNN(coords)

query_ball_point(coord, radius)

query_ball_tree(other, radius)

query_k_nearest(coords, k=1, return_dists=False)

query_nearest_node(coords)

exception syconn.utils.skeleton_utils.NonSimpleNML

Bases: exceptions.Exception

class syconn.utils.skeleton_utils.ShortestPathSegment

Class representing a shortest path between two nodes in an annotation.

from_annotation(annotation, from_node, to_node)

Parameters:

• annotation (EnhancedAnnotation object) – Annotation in which the path is defined
• to_node (from_node,) – First and last node in shortest path, respectively.

from_path(path, annotation=None)

get_node_at_distance(dst, exact=False)

Return node at a specified distant from segment start.

Parameters:

• dst (positive float) – Minimum distance, in physical units
• exact (bool) – If True, generate and return a new node that is at exactly the distance dst (rounded to integer coordinates). The new node will also be added to the ShortestPathSegment.

Returns:

• distant_node (SkeletonNode object) – Node at minimum distance dst from segment start node. If no node is sufficiently distant, return most distant node. This will happen even if exact=True is set!
• total_distance (float) – Distance of distant_node from segment start.

get_subsegment(start_node, end_node)

Return the subsegment of this segment that lies between two nodes.

Parameters:

• start_node (SkeletonNode object) – Node that starts the subsegment. Must come before end_node in the order of the segment.
• end_node (SkeletonNode object) – Node that ends the subsegment. Must come after start_node in the order of the segment.

Returns:

subsegment – Nodes from start_node to end_node, including them, in order. Empty if there is no subsegment limited by the specified nodes.

Return type:

ShortestPathSegment instance

iter_edges()

length()

Return total length of segment over all nodes in physical units.

syconn.utils.skeleton_utils.annoToKDtree(annotations)

Uses scipy kd-trees. Scaling must be set. Only nodes are added to the tree, no edges currently. Creates one kd-tree for every annotation.

syconn.utils.skeleton_utils.annoToNXGraph(annotations, merge_annotations_to_single_graph=False)

Creates a weighted networkx graph from an annotation. Weights are the euclidian distance between nodes. Node objects are preserved. The annotation.scaling parameter is required.

syconn.utils.skeleton_utils.annosToKDtree(annotations)

Uses scipy kd-trees. Scaling must be set. Only nodes are added to the tree, no edges currently. Inserts many annotations into a single kd-tree.

syconn.utils.skeleton_utils.annosToNMLFile(annos, filename)

syconn.utils.skeleton_utils.annotation_from_nodes(nodes, annotation=None, preserve_scaling=True, connect=False)

Take an iterable of nodes and return an annotation that contains those nodes. If the annotation key word parameter is specified, return an annotation in which the nodes are connected as in annotation. Otherwise, the nodes will not be connected.

Parameters:

• nodes (Iterable of SkeletonNode objects) –
• annotation (SkeletonAnnotation instance or NoneType) –
• preserve_scaling (bool) – Whether to copy the scaling from annotation to the new annotation.
• connect (bool) – Whether to copy the connectivity from annotation to the new annotation.

Returns:

new_anno

Return type:

SkeletonAnnotation instance

syconn.utils.skeleton_utils.annotation_matcher(annotations, spotlightRadius=400, samplesize=400, thres=0.25, visual_inspection=False, skip_if_too_small=True, write_match_group_nmls='', write_to_file='')

Algorithm to group annotations that might be the same neuronal structure. Based on a spatial search of a subpopulation of nodes of each annotation in all other annotations. Uses the scipy.spatial kd-tree implementation of fast spatial searches and networkx connected components for the grouping of the annotations.

Parameters:

• annotations (list of NewSkeleton.annotation objects) –
• spotlightRadius (int [nm] that defines the radius of a sphere around a) – currently searched node used to find nodes in other annotations 500 works well, be careful, not easy to tweak.
• samplesize (int of how many nodes to randomly sample from each) – annotation for the matching; higher numbers slows the matching down, but makes the result more accurate
• skip_if_too_small (bool that defines whether to skip an annotation if it) – has less than samplesize nodes. If False (default), all available nodes will be used.
• thres (float [0,1]; fraction of nodes of samplesize that need to) – have a match in another annotation for both annotations to be reported as probable match partners; only reciprocal matches are currently accepted.
• visualize_matches (bool, uses matplotlib and mayavi for debugging) –
• write_to_file (str of path to filename for the output or empty to not write) – a file

Returns:

groups – biological structure

Return type:

list of lists of annotation objects that are probably the same

syconn.utils.skeleton_utils.annotation_to_nx_graph(annotation)

Creates a network x graph representation of an annotation object. :param annotation: NewSkeleton.annotation object :return: networkx graph object

syconn.utils.skeleton_utils.avg_annotation_inter_node_distance(anno, outlier_filter=2000.0)

Calculates the average inter node distance for an annotation. Candidate for inclusion into the skeleton annotation object.

Parameters:

• anno – annotation object
• outlier_filter – float, ignore distances higher than value

Returns:

float

syconn.utils.skeleton_utils.estimateDisagreementRate(annotations, spotlightRadius)

Looks at all nodes >deg 2 and searches for corresponding nodes in the other annotations.

syconn.utils.skeleton_utils.euclNodeDist(node1, node2)

syconn.utils.skeleton_utils.extract_main_tracings(source_folder, scaling)

Go through a set of NML files, extract the largest annotation and save only that annotation to a new file.

Parameters:source_folder (str or list of str) – If str, glob expression yielding the list of NML paths to look at. If list of str, the list of NML paths to look at. Returns:

• Nothing, writes NML files with the same name as before, except the .nml
• extension is replaced by .main_tracing.nml.

syconn.utils.skeleton_utils.filter_nodes(s, criterion)

syconn.utils.skeleton_utils.genSeedNodes(annotations, reqRedundancy, spotlightRadius)

Returns a set of tuples of seed nodes and tracing radii that can be distributed to tracers. If no seed nodes are returned, no end nodes were found that had less redundancy than reqRedundancy, i.e. the annotation is complete.

syconn.utils.skeleton_utils.genZColumnGridSkeleton()

Generates a nml file with a skeleton grid, based on dense columns in z direction to help further annotations. Each column is an individual tree, labeled with a comment. This allows the annotator to hide columns that are already done.

Configure variables directly in the code, at the top.

syconn.utils.skeleton_utils.genj0126SkelObj()

syconn.utils.skeleton_utils.getAnnoWithMostNodes(skeleton)

Many users submit nml files with multiple trees (annotations) that were created accidentally. This functions checks how many nodes each annotation contains and returns the annotation that contains most. Returns none in case there are trees with more than 1 node in each.

syconn.utils.skeleton_utils.getAnnosByCommentRegex(regex, annotations)

syconn.utils.skeleton_utils.getNMLannos(filename)

syconn.utils.skeleton_utils.getNodesByCommentRegex(regex, annotations)

syconn.utils.skeleton_utils.get_all_comment_nodes(annos)

Returns a list of all nodes with a comment for given annotations.

syconn.utils.skeleton_utils.get_all_node_comments(annos)

Returns a list of strings (all node comments) for given annotations.

syconn.utils.skeleton_utils.get_annotations_with_comment(s, expression, flags=0)

Returns a list of SkeletonAnnotation objects with comments that match expression.

Parameters:

• s (NewSkeleton instance) –
• expression (str) –
• flags (int) – Regular expression flags for re.search()

syconn.utils.skeleton_utils.get_closest_node(location, annotation, cutoff=None)

Return the node in annotation that is closest to location.

Parameters:

• location (SkeletonNode object or list of numeric type) – If SkeletonNode object, node for which to find closest match in annotation. If list of numeric type, coordinate for which to find closest match in annotation.
• annotation (EnhancedAnnotation or SkeletonAnnotation object.) – Annotation in which to search for matching nodes. Spatial lookup will be faster when using EnhancedAnnotation.
• cutoff (float or None) – If float, return closest node only if its distance to location is smaller than cutoff. If None, ignore.

Returns:

closest_node – Closest node to location in annotation or None if there is no node in annotation or closest node in annotation is futher away from location than cutoff.

Return type:

SkeletonNode object or None

syconn.utils.skeleton_utils.get_convex_hull(s, scaling='nm')

Return scipy’s convex hull of the set of points contained in a skeleton.

Parameters:

• s (NewSkeleton instance or str) – If str, path to a file to load skeleton from
• scaling (str) – As in get_node_positions_as_nparray

Returns:

hull

Return type:

scipy.spatial.ConvexHull instance

syconn.utils.skeleton_utils.get_end_nodes(annotation)

Return set of SkeletonNode objects in annotation that only have one neighbor, i.e. that are ends.

Parameters:annotation (EnhancedAnnotation object) – Annotation in which to search for end nodes

syconn.utils.skeleton_utils.get_largest_annotation(skeleton)

Return the physically largest annotation from skeleton.

Parameters:skeleton (NewSkeleton instance) – Returns:

• largest_annotation (SkeletonAnnotation instance or None) – Largest annotation or None if there is no annotation in skeleton.
• largest_annotation_length (float) – Physical length of largest_annotation

syconn.utils.skeleton_utils.get_max_node_id(fname)

Return maximal node ID.

syconn.utils.skeleton_utils.get_nml_str_from_knossos_file(fname)

Take a knossos file and return the NML contents as str, i.e. open the file and read it or extract and read the annotation.xml from a kzip.

syconn.utils.skeleton_utils.get_node_at_position(annotation, position)

Return a node from annotation located at a given position.

Parameters:

• annotation (SkeletonAnnotation instance) –
• position (Iterable of int with length 3) –

Returns:

node – Node at position, or None if there is no node at that position.

Return type:

SkeletonNode instance or None

syconn.utils.skeleton_utils.get_node_positions_as_nparray(s, scaling='raw')

Return the positions of all nodes in a skeleton as a numpy array.

Parameters:

• s (NewSkeleton instance) –
• scaling (str) – if ‘raw’, use the raw node coordinates, otherwise, use scaled coordinates.

syconn.utils.skeleton_utils.get_nodes_with_comment(s, expression, flags=0)

Return a list of SkeletonNode objects with comments that match expression.

Parameters:

• s (NewSkeleton instance or SkeletonAnnotation instance.) –
• expression (str) –
• flags (int) – Regular expression flags for re.search()

syconn.utils.skeleton_utils.get_nodes_with_token(s, token)

Return a list of SkeletonNode objects in s with comments containing token. A token is defined as a substring of the node comment separated by a semicolon and optional whitespace.

Parameters:

• s (NewSkeleton instance or SkeletonAnnotation instance.) –
• token (str) –

Returns:

matching_nodes

Return type:

list of SkeletonNode instances

syconn.utils.skeleton_utils.get_nonempty_annotations(s)

syconn.utils.skeleton_utils.get_reachable_nodes(node)

Return a set of all nodes that are reachable from node (part of the connected component that includes node).

Parameters:node (SkeletonNode instance) – Returns:reachable_nodes Return type:set of SkeletonNode instances

syconn.utils.skeleton_utils.get_subsegment_by_distance(annotation, start_node, end_node, near_dst, far_dst)

Return ShortestPathSegment instance corresponding to a subsegment delimited by physical distances of a subsegment delimited by start and end nodes.

Parameters:

• annotation (EnhancedAnnotation instance) –
• start_node (SkeletonNode instance) – Start node, must be contained in annotation.
• end_node (SkeletonNode instance) – End node, must be contained in annotaton.
• near_dst (positive float) – Distance of subsegment start seen from start_node, towards end_node.
• far_dst (positive float) – Distance of subsegment end seen from start_node, towards end_node. Must be larger than near_dst.

Raises:

Exception – If near_dst is larger than far_dst

Returns:

subsegment

Return type:

ShortestPathSegment instance

syconn.utils.skeleton_utils.get_the_nonempty_annotation(s)

Convenience function that returns the one nonempty annotation from a NewSkeleton instance s and raises an Exception if more than one exist in s.

Parameters:s (NewSkeleton instance) –

syconn.utils.skeleton_utils.get_the_nonempty_connected_annotation(s)

Convenience function that returns the one nonempty, singly connected annotation from NewSkeleton instance s and raises Exception if more than one non-empty annotation exists or if that one annotation is not singly connected.

Parameters:s (NewSkeleton instance) –

syconn.utils.skeleton_utils.has_3_4_worktime_bug(filename)

Knossos 3.4 introduced obfuscated (xor’ed) worktimes. When skeletons were created in versions prior to 3.4 and went through multiple load-save cycles in 3.4, the work time would alternatingly be xor’ed and not xor’ed and the node timestamps became unreliable.

This works on the assumptions that times are not larger than ca. 15 days, so we can use the high bits of the xor key to detect whether a time has been xor’ed or not.

This function returns True if the work time in the file can not be trusted.

filename is the path to the nml file.

syconn.utils.skeleton_utils.has_node_id_overflow_problem(fname, cutoff=2147483647)

Due to a knossos issue, node IDs can become excessively high. This checks whether a file is affected.

Parameters:

• fname (str) – Path to nml or kzip
• cutoff (int) – Start warning from this node ID

Returns:

problem – True if there is a problem

Return type:

boolean

syconn.utils.skeleton_utils.has_node_id_overflow_problem_dir(path, expression='*', cutoff=2147483647)

Like has_node_id_overflow_problem, but works on a directory.

Parameters:path (str) – Path to directory of NMLs and / or .k.zip Returns:

• problem (list of str) – List of paths to problematic files
• no_problem (list of str) – List of paths to intact files

syconn.utils.skeleton_utils.is_simple_skeleton(s, fail_on_empty=False)

Returns the annotation if the skeleton contains exactly one non-empty annotation. If fail_on_empty is True, return the annotation only if there is exactly one non-empty annotation and no empty annotations. Return None otherwise.

syconn.utils.skeleton_utils.is_singly_connected(annotation)

Return True if all nodes in annotation are connected, False otherwise (including if there are no nodes).

Parameters:annotation (SkeletonAnnotation object) –

syconn.utils.skeleton_utils.iter_nodes_dfs(annotation, root)

Iterate over the nodes in annotation, starting at root, using depth-first search.

Parameters:

• annotation (SkeletonAnnotation object) –
• root (SkeletonNode object) – Must be contained in annotation.

syconn.utils.skeleton_utils.iter_nodes_regex(fname)

Iter over nodes in a NML / kzip using regular expressions. Much faster than XML parsing.

Parameters:fname (str) – Path to nml or kzip

syconn.utils.skeleton_utils.load_j0251_nml(path_to_file, merge_all_annos=False)

syconn.utils.skeleton_utils.load_j0256_nml(path_to_file, merge_all_annos=False)

syconn.utils.skeleton_utils.load_jk_NML(pathToFile, ds_id, scaling, dataset_dims, remove_empty_annotations, merge_all_annos=False)

Loads a NML file and add some specific attributes to the returned skeleton annotation objects.

syconn.utils.skeleton_utils.loadj0126ConsensusNMLsInDir(directory)

Loads all .nml files inside a directory and give warnings.

syconn.utils.skeleton_utils.loadj0126NML(path_to_file, merge_all_annos=False)

syconn.utils.skeleton_utils.loadj0126NMLbyRegex(regex)

syconn.utils.skeleton_utils.merge_annotations(a1, a2)

Take all the nodes and edges from annotation a2 and add them to annotation a1. Annotation a2 remains intact after calling this function.

Parameters:a2 (a1,) – Returns:

• a1 (SkeletonAnnotation instances) – Same annotation that was passed as a parameter, modified “in place”.
• As an additional side effect, this function sets the old_to_new_nodes
• attribute on a1, which is a dict mapping nodes from a2 to nodes in a1
• after the merge.

syconn.utils.skeleton_utils.neighbour_next(node)

syconn.utils.skeleton_utils.nodes_to_NX_graph(nodes)

Takes an iterable of nodes and creates a networkx graph. The nodes must be part of a skeleton annotation object. This is useful to get the path length of a subset of a skeleton annotation, defined by nodes. :param nodes: iterable of SkeletonNode objects :return: networkx graph

syconn.utils.skeleton_utils.nx_graph_to_annotation(G, scaling=None)

Turn a NetworkX graph into a SkeletonAnnotation. Nodes in the graph are assumed to be SkeletonNode instances, but only the edges present in the NetworkX graph are used, potentially existing NewSkeleton style edges are ignored.

Parameters:G (NetworkX graph) – ... where the nodes are SkeletonNode instances Returns:a – Fresh SkeletonAnnotation, with newly created SkeletonNode objects Return type:SkeletonAnnotation instance

syconn.utils.skeleton_utils.prune_short_end_branches(anno, length, debug_labels=False)

Remove all end branches that are shorter than length.

Parameters:

• anno (SkeletonAnnotation instance) –
• length (float) – In physical units, up to which length to prune
• debug_labels (boolean) – Whether to add comments to the pruned skeleton for debugging purposes. Will label identified end nodes and upstream nodes.

syconn.utils.skeleton_utils.prune_stub_branches(annotations, len_thres=1000.0, preserve_annotations=True)

Removes short stub branches, that are often added by annotators but hardly represent true morphology.

Parameters:

• annotations –
• len_thres –
• preserve_annotations –

Returns:

syconn.utils.skeleton_utils.remove_all_node_comments(anno)

syconn.utils.skeleton_utils.reset_nml_for_heidelbrain_taskfile(f, time=0, current_knossos='3.4.2')

Takes an NML and applies some corrections to make it suitable as a regular Heidelbrain task file. That is, timestamps are removed or reset and the version strings are set to a recent version of knossos. The file at f is altered by this function.

Parameters:

• f (str) – Path to file
• time (int) – What to set the time to, in milliseconds. This can be used to add gratuitous time to a heidelbrain task.
• current_knossos (str) – Knossos version to set createdin and lastsavedin to.

syconn.utils.skeleton_utils.return_process(node)

syconn.utils.skeleton_utils.save_annotations(annotations, fname, scaling=None)

Save annotations to an NML.

Parameters:

• annotations (Iterable of SkeletonAnnotation instances) –
• fname (str) – Filename to save to
• scaling (Iterable of float or None) – Scaling parameter to set on the NML file.

Returns:

Return type:

Nothing, writes file.

syconn.utils.skeleton_utils.setAnnotationStats(annos)

Sets the following stats: anno.pathLen in um anno.numBranchNodes

syconn.utils.skeleton_utils.shortestPathBetNodes(annotation, node1, node2)

Shortest path length between 2 nodes in units of scaling / 1000

syconn.utils.skeleton_utils.skeleton_from_single_coordinate(coordinate, comment=None, branchpoint=False)

Return a NewSkeleton object containing exactly one annotation with exactly one node at position coordinate. This is good for generating seed NML files for specific coordinates.

Parameters:

• coordinate (Iterable of int) –
• comment (str) –
• branchpoint (bool) – Whether this node is a branchpoint.

Returns:

s

Return type:

NewSkeleton instance

syconn.utils.skeleton_utils.split_by_connected_component(annotation)

Take an annotation and return a list of new annotations that represent the connected components in the original annotation.

Parameters:annotation (SkeletonAnnotation instance) – Returns:connected_components Return type:set of SkeletonAnnotation instances

syconn.utils.skeleton_utils.write_anno(a, out_fname)

Write SkeletonAnnotation to file.