Manipulators¶
Hardware control¶
Manipulators are groups of motorized axes, typically an XY stage or an XYZ unit.
The basic class is a ManipulatorUnit
, which depends on a controller. The controller is
a set of axes that can be moved independently (for example the Luigs and Neumann controller, which
controls up to 9 axes).
Example:
controller = LuigsNeumann_SM10()
stage = ManipulatorUnit(controller, [7, 8])
A ManipulatorUnit
can be moved with relative or absolute displacements expressed in µm.
Calibrated units¶
Calibrated units are manipulator units that can be moved in the coordinate system of the camera, called
the reference system.
A CalibratedUnit
must be associated to a camera, a microscope Z axis, and can be attached to an XY stage.
A CalibratedStage
is a special kind that a horizontal XY stage (i.e., to be parallel to the focal plane of the
microscope).
Examples:
calibrated_stage = CalibratedStage(stage, microscope=microscope, camera=camera)
XYZ = CalibratedUnit(unit, calibrated_stage, microscope, camera=camera)
The position in the camera system is given by \({\bf M}\cdot{\bf u} + {\bf r}_0 + {\bf r}_S\), where \({\bf u}\) is the position of the manipulator axes, \({\bf r}_S\) is the position of the stage in the reference system to which the manipulator is attached, \({\bf M}\) is coordinate change matrix and \({\bf r}_0\) is an offset.
Movement algorithms¶
Reference move¶
The basic move is a reference_move
. It simply inverts the matrix relation to find the
target position in the coordinate system of the manipulator coordinates.
However, this is not as simple as it sounds. For some manipulators (including Luigs and Neumann),
the resulting displacement is not necessarily a straight line because each axis has a fixed speed independent
of the movement. This can result in a broken trajectory; first a diagonal move then a move in the remaining
directions. As a result, the pipette could collide with the coverslip or other problems.
To avoid this problem, the method has a safe
option. If True, the method first determines whether the
the third axis, which is assumed to be Z, will be moved up or down (see calibration algorithms).
If it is going up, then this axis is moved first; otherwise it is moved last. This simple algorithm
maximizes the minimum altitude of the trajectory, so as to avoid colliding with the coverslip.
This is only done with absolute moves and not relative moves.
Withdraw¶
The withdraw
method moves the first axis to its upper endpoint. This presupposes that the two endpoints
have been previously identified.
Focus¶
The focus
method moves the microscope Z axis so that the tip is in focus. This does not use an autofocus
but rather the calibration system (so the manipulator must be correctly calibrated for this to work).
Safe move¶
The safe_move
method moves the manipulator to a target point, with a trajectory that aims at minimizing
mechanical interaction with the tissue. It also essentially removes the pipette from the field of view during the
approach, which could be helpful if tracking the cell.
If the movement is up, a normal movement is done (with the safe
option). If it is down, then the trajectory
is more complex. First, the manipulator is moved horizontally, then along the first axis of the manipulator.
Note that by horizontally, it is meant that the start and end positions are on the same horizontal plane, but
the trajectory does not necessarily remain in that plane for reasons explained above; thus the
safe
option is also used.
If the recalibrate
option is True
and the movement is at least 500 µm, then the program tries to fix
errors in calibration before the target. To this end, the manipulator stops 50 µm before the target,
then focus on the tip, automatically recalibrate (see below), then move the focus back, and finish the movement.
Moving a new pipette in the field¶
This is not fully tested code. The move_new_pipette_back
method moves a new pipette into the field.
This assumes that the calibration is right, except for an offset (due e.g. to the length and slightly different
geometry of the pipette). The algorithm is as follows:
Move the pipette 2 mm before target position (in the direction of the first axis), which is the center of the microscope view.
Take 10 photos at 10 Hz.
Calculate the mean standard deviation of the images (more or less the contrast).
Move the pipette down by 100 µm along the axis.
If the standard deviation of the image differs by at least 20% from the mean calculated previously, stop.
Otherwise, go to 4; stop at 5 mm.
In practice, if the pipette is cleaned, this method might not be that useful.
Move and track¶
The move_and_track
is used by calibration algorithms. It moves the pipette along one axis,
then focus the microscope on the tip using calculation and then template matching. Optionally,
it also moves the stage to center the tip. The final image is focused on the tip, but the tip
is not necessarily in the center (depending on the precision of calibration).
Finally, it returns the position of the tip on screen and focal plane.
Move back¶
The move_back
method is used by calibration algorithms.
It moves the microscope, manipulator and stage to a given position (previously stored), in
a certain order that is intended to avoid collisions.
First, the microscope is moved (normally, up), then the manipulator, then the stage.
The pipette is then back at the initial position, which is supposed to be in focus in the center
of view. Then the pipette is located and refocused, and the pipette position and focal plane are
returned.
Calibration algorithms¶
Calibration consists in determining the matrix \({\bf M}\) and the offset \({\bf r}_0\), as well as whether the axes go up or down (in Z) in the positive direction.
Recalibration¶
This assumes that the manipulator is correctly calibrated, except for an offset.
The method recalibrate
updates \({\bf r}_0\) assuming that the tip is in the center of view
(red cross), or at the given (x,y) position on screen if provided (right-click on the standard
interface).
Stage calibration¶
The stage is assumed to be horizontal, and thus the Z axis of the microscope is not moved. It is assumed that there is an object in focus in the field of view, attached to the stage (pipette, or coverslip). Algorithm:
Take a photo of the center of the field: this is the template.
Move the first axis by 40 µm, and locate the template in the image: deduce the first column of \({\bf M}\).
Repeat for the second axis.
Using the first estimate of \({\bf M}\), move to each of three corners of the image (top left, top right, bottom left), with a safety margin, and locate the template.
Calculate \({\bf M}\) again based on these three points.
Manipulator calibration¶
This is the calibrate
method, plus a number of methods that it calls.
The tip must be in focus at the center of view.
Initial steps
Calibrate the stage to which it is attached.
Take photos of the pipette along the Z axis of the microscope, every 1 µm over distance
stack_depth
(positive and negative).
First estimate
Move and track the first axis by a distance equal to half the
stack_depth
. As initially the matrix is zero, there is no predictive move of the focus.Repeat for each axis.
Calculate the matrix.
Go back to the initial position.
This first very crude estimate is used to calculate the vertical direction of the axes.
Up directions
This is done in method calculate_up_directions
. It takes the matrix and estimates
for each axis whether a positive movement makes the pipette go up or down.
Then the minimum reachable Z (coverslip) is determined as 300 µm below the current position,
unless it has been specified explicitly (floor position).
Calibration
Each axis is calibrated in turn. For each axis:
Double the movement amplitude.
Check whether the movement is reachable (which presupposes that ranges have been set).
Estimate whether the movement will make the pipette move out from the field of view.
Move the pipette and track, and move the stage to compensate if the pipette is out of field.
Calculate the relevant column of \({\bf M}\), based on camera positions before and after the movement.
Repeat
calibration_moves
times.Move back to the initial position.
Calculate the relevant column of \({\bf M}\), based on camera positions before and after the movement.
Thus, only the last movement (which is the largest one) is actually used to calculate the matrix.
Manual calibration¶
The manual_calibration
method takes 4 points chosen by the user, and deduce the matrix
from them.
Automatic recalibration¶
Locate the pipette over a depth of +-25 µm, using templates and movements of microscope Z.
Update the offset \({\bf r_0}\) (recalibration).
With option
center
, move the stage and focus so that the pipette tip is centered.