Visual Detection and Tracking Methods for E. superba (Antarctic Krill)

Antarctic krill are a keystone species of the Southern Ocean. They have been well documented over large spatial scales but generally not quantfiably at the scale of single individuals in the open water column. It is important to study how individuals behave in their natural environment in order to further understand how they interact within dense krill aggregations. Using a pair of calibrated grayscale stereo cameras mounted on a towed instrument sled, krill were imaged in situ at 10Hz in the bays along the western Antarctic Peninsula during austral winter 2013. Krill were identified and tracked through the images using a newly developed identification and tracking method that collates krill motion properties such as distance traveled, velocity and track duration using image processing techniques. Stereo geometry was used to define the krill motion data in the camera coordinate system and define the overall imaging volume to be approximately 2.0m. The tracking method performed successfully for 60 − 80% of tracks in a sample set of images. Difficulties in tracking krill successfully included excessive sled motion (heave), krill swarming (or schooling) behaviors and rapid changes in krill motion not accounted for by the tracking algorithm. An analysis of the krill velocities found that krill generally swam at less than 1m/s and increased to 2m/s while aggregating. This new imaging system successfully tracked and identifed krill in the midwater column and can be used to generate large motion data sets to better inform Antarctic krill behavioral and circulation studies.

Two examples of the differences in krill motion (for two unique krill) relative to the camera sled versus relative to the water. . . 20 3 Basic krill track statistics from four dives (14, 15, 16 and 23).  Each camera has an approximate 1.0m tall by 1.2m wide field of view in focus to approximately 1.5m from the camera. . . . . 6 5 An initial gray-scale krill image from the stereo pair which has been initially filtered with a high-pass filter to increase edge contrast around the krill and remove any light patterns. . . . . 7 6 An example of the image processing that identifies possible krill (a-c is to be submitted for publication.

Introduction
Antarctic krill, Euphausia superba, play a prominent role in the Southern Ocean as the keystone species of the Antarctic ecosystem [2,3,4]. They are greatly affected by the most prominent factors of climate change which are ocean acidification and sea-ice retreat. Additionally, krill are major consumers of phytoplankton which are responsible for most of the carbon fixing in the upper ocean [5,6]. Given these important roles it is necessary to understand krill behavior in order to further understand Southern Ocean ecology and implement better ecosystem management practices [2,7,5]. food supply, making it one of the richest krill spawning regions in Antarctica [8].
Krill distributions and abundances have been typically characterized over broad spatial areas using traditional methods such as ship-based acoustics and net tows, both of which tend to aggregate behavior [2]. Krill are infrequently studied at the scale of individual animals. Krill display significant spatial and temporal variability due to their life cycle migration patterns and swarming behaviors [3]. These behaviors impact other larger zooplankton species by providing or competing for food sources [10,2]. In order to fully understand Antarctic krill ecology and how they impact their environment, small-scale behavioral data at the individual level needs to be further quantified and compared to larger scale observations [2,4,7].
Studying krill (and other zooplankton) behavior using a net-mounted camera is also limited by the documented net avoidance behavior of euphausiids [9,18].
In an effort to further methodologies for studying zooplankton behavior in situ, this paper describes the detection, identification and quantification of krill motion in the water column using image data from a midwater camera system. In situ imaging can offer behavioral and three dimensional motion data that traditional sampling methods cannot, while providing insight into the validity of laboratorybased experiments. To detect and track krill through collected images, a general method was devised to identify krill based on their shape and aspect ratio, and then track individual animals through image sequences. With this approach krill were succesfully identified and tracked forward and backward in time to create a database of motion tracks and associated data such as speed, aspect ratio and size.
Image data were collected using a stereo camera sled during the May -June 2013 Seasonal Trophic Roles of Euphausia superba (STRES) research cruise aboard the RVIB Nathaniel B. Palmer in the bays along the WAP. The camera sled was deployed 40 times at the locations highlighted in Figure 1. Additional MOCNESS data were collected and analyzed during this cruise to identify zooplankton taxa and abundances at the dive sites [19]. Initial krill distributions using image data from the camera sled were examined as well [11]. The austral winter season pro- vided a low turbidity water column to image the krill [9,3,20,4]. In general, only E.superba were collected in the net tows and observed by the cameras.

Materials and Methods 1.2.1 Camera Sled and Image Data Acquisition
The camera sled system ( Figure 2) was designed and built for the purpose of imaging biota (1cm -6cm in length) in the water column. The sled is rated to 2000m water depth and offers the ability to observe the water column in real time on the ship. The sled system contains a stereo pair camera, Sea-Bird 49 FastCAT CTD, a 1200kHz ADCP, a lightfield camera and LED lights (red and white) to illuminate the field of view ( Figure 2a) [21,9,17].
The sled can be used to image in side-looking or down-looking orientations   : Each camera has an approximate 1.0m tall by 1.2m wide field of view in focus to approximately 1.5m from the camera.

Krill Identification
Krill were identified in each image using the following series of processing techniques implemented in MATLAB ® .  fications. The aspect ratio for a krill should be greater than three. The krill size should be greater than 400 pixels ( Figure 6c) [11].
6. Each positively identified "krill" in an image is indexed and an initial krill count per image is produced. The accepted krill are highlighted in green over the original gray-scale image (Figure 6d).
An elliptical shape was chosen to open the binary image due to a krill's generally tapered elliptical (two-dimensional) shape [22,21,25,23]. To verify that the segmentation algorithm was successfully identifying krill, a comparison between a traditional human krill count and the automated count was completed. One hundred random images containing krill from the same dive were chosen and manually counted. The algorithm was then run with two differently sized structuring elements to identify krill. The two automated datasets and the human visual count are compared in Figure 7 with a linear regression describing the agreement between the algorithmic and traditional counts.

Krill Tracking
Using krill properties provided by the identification step, krill can be tracked between image frames. The general automated krill tracking method is described below.
1. An initial krill to track is chosen from the image data from the initial segmentation algorithm with the constraints that it has an aspect ratio greater than three and a size greater than 400 pixels [11]. All krill selected satisfy these requirements.
2. This krill is given an identifying global krill number to flag it through the adjacent images forward and backward in time.
3. In the image sequence, possible matches are searched for within a set radius (100pixels) from the krill's position in the prior image. Krill, on average, did not travel more than 100 pixels between image frames when swimming in a steady manner. 4. For all possible matches motion vectors relative to the initial krill positions are calculated. These vectors are compared to an estimate of the krill's prior motion, if available. A trajectory angle for each possible match is calculated and used to calculate the vector dot product to help relate possible matches [26]. If this is the first krill in a track, a motion vector cannot be calculated but a vector dot product is for all possible matches to help relate them to the initial krill.

Synthesis of Krill Motion Data
An organizational structure for the krill motion data is created after tracking krill through an image directory. The tracking method produces three data  trajectory angle (θ)).

A tracks array flags (using binary identifiers) when a krill is present in an
image or not.
3. An image list is a list of image names used by the krill tracking algorithm.
An image directory can be visualized ( Figure 9) and individual track data can be referenced by image number and global krill number (Table 1)

Krill Camera Geometry: Relative Velocities
In order to analyze krill motion data in a euclidean reference frame, rather than in pixel coordinates, the data are transformed using the stereo camera geometry and triangulation methods detailed in [1,23,28]. To generate the stereo information a tracked krill in the left camera must be correctly matched in the right camera image.
This correspondence problem is solved by rectifying the stereo images following procedures described in [23]. The intrinsic and extrinsic properties of each camera are known through a calibration done prior to fieldwork [29]. An image rectification transform to the left and right images makes them co-planar (or rather, horizontally collinear) to one another. Once the images are co-planar, the epipoles in each are sent to infinity (Appendix B.1) making the epipolar lines collinear to one another.
Once rectified, matching krill between image pairs becomes a 1-D search across the adjacent image rows [1,23,26]. A normalized sum of squared differences (SSD) is used to match krill between corresponding left and right images. Images are normalized (I norm ) before performing the SSD by where I is the average image intensity. The SSD is calculated as, where a square 40x40 pixel template (T ) is created centered on a krill's position

Water Relative Velocities
Krill velocities are computed using their distance traveled (in meters) from stereo triangulation and track duration (in seconds). Calculated velocities are relative to the camera, and not the surrounding water (uncorrected V ave , Table   2). To determine water relative krill motion, water velocities measured with the ADCP were compared to the krill velocities at the same time ( Figure 13). The calculated krill motion data and ADCP water velocity data from the first range bin (2.5m) are matched using time stamps and then oriented using the instrument's coordinate frame (Figure 2a). In general the magnitude of ADCP water velocities are small (0.2m/s), relative to the observed krill velocities ( Figure 13). Along track velocities can be corrected to produce water relative measurements using the ADCP water velocity data. This is discussed in section 1.3 where resulting krill position and velocity trends are examined.  Table 2: Two examples of the differences in krill motion (for two unique krill) relative to the camera sled versus relative to the water.
Krill velocities and tracks can be shown in both two dimensions and three dimensions ( Figure 14). The three dimensional tracks (Figure 14b) also highlight the variability in krill positions in the z direction which is most likely due to triangulation error [1,23,28].

Results
To test the performance of the tracking algorithm four dives were analyzed.
This includes 17 depth horizons encompassing daytime and nighttime periods.
Approximately 4000 krill were tracked. In general, tracking worked best for images with less than 20 krill.  This trend is demonstrated in the overall dive statistics with an average track length of 0.05m, velocity of 3.71m/s and a duration of 4.05sec (Table 3).

Automated Tracking Performance
To assess the errors in more detail dive 15 was chosen to estimate what percentage of tracks are successful versus incomplete for a set of images. This dive was chosen since it has the greatest number of uncorrupted depth horizons, meaning ship heave or pervasive krill swarms did not dominate the majority of image data, out of the entire dataset. An incomplete, or bad, track is defined as a continuous motion trajectory that has been split into two or more tracks, or a track populated by bad matches (Figure 15). This means they are identified in the track data library with multiple global krill numbers resulting in over counting and er-roneously short tracks. A good track is defined as a continuous motion trajectory that starts with the krill appearing in an image and ending when it leaves the frame or becomes too small to track.    To determine whether any of the bad tracks could be fixed they were split into two categories: corrupted and broken (Table 4b and   4c).
"Corrupted" meant the track wasn't capable of following the animal and "bro-  (Table 4b and 4c). For the random track set a larger percentage of broken tracks (38%) were identified manually than with automated checking (21%). The percentage of broken tracks that could be fixed manually or automatically is comparatively similar at 9% and 2%, respectively (Table 4b and   4c).
In total this sample suggests that 60% to 80% of tracks ( Figure 17 and Table   4a) are successful (or complete) for a given image set. This estimate can be applied to a general track count per image ( Figure 17).

Tracking Compared to Identification Performance
There are several differences between the initial segmentation techniques to identify krill in an image and the automated krill tracking that associates potential krill with motion tracks. Segmentation uses only information in a single image to identify possible krill. This requires that the criteria to determine a krill needs to be restrictive enough to avoid false positives. The automated tracking process uses a two-step krill identification. First, krill are selected to start a track based on the same criteria as the initial krill identification method, using the more restrictive   To compare the number of krill identified using the automated tracking algorithm in comparison to the initial segmentation techniques depth horizon #14 from dive 15 was used. Krill counts from tracking and segmentation were compared and it was found that the automated krill tracking identifies more krill than using segmentation alone ( Figure 17). The number of good tracks per image were estimated using the successful track estimate (60% -80%) from the earlier tracking performance assessment (Table 4). This result can also be seen in example images showing the krill identified with the segmentation and tracking methods ( Figure   18).

Krill Position and Velocity Trends
The stereo camera geometry is able to derive krill positions in 3D and the associated velocities ( Figure 14). The resulting position data has noticeable variability along the z-axis (Figure 14b).
By plotting a random set of 20 velocity tracks and subtracting the average velocity for each direction (Figure 19), it is clear that V z has the greatest variability.
ADCP water velocties were plotted for the same depth horizon to determine how much the observed water velocties were contributing to krill motion ( Figure 13).
The ADCP velocties were found to be small suggesting that water velocities are not significantly contributing to krill motion variability ( Figure 19). This velocity variability in the z-direction is most likely due to the error associated with triangulating positions during the stereo reconstruction. Triangulation in pixel location has the weakest constraint in z ( Figure 19) and is easily effected by small errors when matching krill centroids between images [23,1,28].   [30,31]. Therefore it is unlikely that any velocities over 2m/s are accurately reporting krill motion and that the high percentage of velocity outliers (Table 5) are forcing the median and average krill velocity values higher than they actually are. The horizon velocity distributions agree with this with the exception of horizon #14 which most likely has a higher average velocity and similar velocity values.
Velocity distributions are more closely examined by looking at velocities for horizon #13 ( Figure 21). As the population of krill increases over time the more erratic (and higher) the velocity values. It can be concluded that the tracking method is performing better in lower krill population environments, calculating more accurate along-track velocities, than in high population environments.   (Figure 20b) are tabled for dive 15. Horizon #14 had the greatest number of velocities tracked but the lowest percentage of outliers.

Discussion
Tracking and quantifying krill motion data in situ in the open water column is an evolving problem that is not without error. Reconstructing tracks in three dimensions using stereo geometry and triangulation methods revealed that poor depth estimates in the camera relative z-direction and produces errors when calculating krill positions and velocities in three dimensions (Figure 14 and 19). Error in z is most likely from two sources: triangulation and bad stereo matching. Depth The automated tracking algorithm successfully tracks 60% to 80% of krill in an image (Table 4 and Figure 17) using a discrete, two-step tracking method (Figure 10). A traditional linear model-based approach, such as a Kalman Filter, was not used as krill motion is generally non-linear [32]. Krill do not exhibit linear, predictable trajectories when swarming, stalling or rapidly accelerating that can be reliably modeled [3,20,10,11,2]. The automated tracking method strives to account for these krill behaviors by estimating position based on the prior krill motion vectors along-track and considering all possible matches within an empirically determined search radius of the estimated new position (Figure 14).
More robust depth triangulation would aid in tracking krill in three dimensions. This would also improve the krill velocity calculations (Figures 14 and 19, Tables 2 and 3), and existing observed and modeled krill velocities used in behavioral studies [11,3,20,2,7,12,25,13,19]. Other improvements to the matching include creating a variable-sized krill template that conforms to the size of each krill when solving the SSD and adding range constraints to increase the robustness of the SSD [1,23,28]. Also a higher camera sampling rate than the current 10Hz should be considered in instances of high krill populations. Better tracking in dense aggregations would increase the amount of available krill behavioral data in swarms that potentially impact surrounding physical ocean processes [3,9,33].
Additionally, krill velocity ensemble statistics further confirm that the tracking method more accurately observes krill in low-population environments (Figures 20   and 21). Velocities observed in lower population images corroborate earlier work [10,30,31]. The tracking method can now be used to generate large motion data sets to further study behavioral patterns of krill. In turn, these can better inform Antarctic circulation study parameters since the WAP is one of the most affected regions by seasonal sea ice flows which provides a crucial habitat and survival mechanism for krill larve and juvenile communities as discussed in [34,35,8,30].
Ecosystem management and conservation policies would also greatly benefit from a better understanding of krill behavior in the Southern Ocean as described in [5].
The image data collected by the camera sled instrumentation system combines traditional sampling methods (CTD and ADCP) with stereo camera image data. This combination has the potential to compare krill track data directly to acoustic krill data ( Figure 22) [36,21,2,25,37]. It improves upon the currently available imaging systems (both in situ and in the lab environment) and generates quantitative physical and behavioral zooplankton data [3,10,13,38,39,14,27].
The camera sled system can be deployed for future studies of Antarctic krill or other zooplankton species.

Conclusions
In conclusion, the automated krill tracking algorithm can track krill in situ in both two dimensional camera relative units and in three dimensions using stereo reconstruction. The automated tracking algorithm and track library are tools that provide detailed access to the behavior of individual animals. In situ track data helps refine and expand the available methods to study krill motion and behavioral patterns.  • Ellipses are rotationally invariant.
• Prior camera configuration (calibration and focus) helped estimate the ellipse size needed to identify krill.
• A structuring element collates region properties (area, major and minor axes) for each identified krill (step # 4) [22,23]. The decision tree that the krill tracking method uses to determine matches between image frames. If a krill has a prior track, the average and standard deviations for several recorded variables are used to choose the matching krill.