Application Focused Image Sensors for Industrial Barcode Scanners
Author: Gareth Powell, Senior Marketing Engineer, Teledyne-e2v.
With the constant drive for better and cheaper cameras for industrial imaging applications, cmos image sensors can make use of unique system on chip (soc) features. It is possible; thanks to 3D chip stacking and back-side illuminated (BSI) cmos image sensors to integrate many image processing tasks into a single device. The future will see sophisticated machine learning and dedicated intelligence computed on-chip while simultaneously grabbing images for compact, higher speed computer vision systems.
However, chip thermal management and power distribution are current obstacles that this new large-scale integration technology has to overcome.
In the meantime, advanced front side illuminated (FSI) cmos sensors are integrating analog and digital features to provide economic and performance oriented solutions. The key to achieving these objectives comes from judicious partitioning of what is useful for system performance aspects and/or cheaper to embed in the image sensor soc. Here the imaging system application and the role of existing image processing devices such as CPU, FPGA or DSP has to be highly considered because redundancy offsets the economic concept. Creating a standard imager soc product that provides its target application market with a viable solution that is able to reach high quantity production (and therefore achieve lowest production cost) requires intense dialogue between the two communities.
Main market players hardware, software, system architects and optical engineers, that are designing ever evolving end camera products that target a specific application segment, and a multi-disciplined, Image Sensor development team, to find innovative product solutions by benefitting from their respective technology/application knowledge, between semiconductor technology and final camera product.
This article will present an example of an application focused cmos image sensor family for use in barcode scanning and other embedded vision applications, with examples of their use cases, and some future perspectives.
Barcodes – an overview of the most pervasive of coding systems, and associated reading technology.
With the holiday season just behind us, and the seasonal peak in consumer retail/on-line spending, logistics and transport, manufacturing and distribution, etc. well in excess of 5 billion barcodes are scanned per day. Considering that the first barcode was scanned in the 70s on a packet of chewing gum, it is clearly a formidable method of providing a machine-readable code, which is still growing in deployment into new fields of application. A 1D barcode symbol that resembles the first to emerge (but printed smaller and smaller as label printing technology advances) represents still the majority and is used for UPC (unique product code) applications in retail, transportation and logistics, etc. The 2D barcode emerged in its various forms to offers a significant increase in code-able data versus the traditional 1D – typically 1D barcodes can have from 20-25 characters while 2D codes go beyond 2,000 characters depending on the specific barcode type. Apart from the obvious ability to store greater product info and details, check-sum and other error correction techniques can be coded in 2D barcodes to ensure increased tolerance to poorly printed or damaged codes. 2D barcodes are used heavily in some market segments such as for direct part marking (DPM) onto individual components used for automated production in the manufacturing segment.
The 2D barcode reader technology transition began around 15 years ago but now represents the majority of today’s market since it can also read existing 1D codes in addition to 2D symbols.
Not all 2D barcode scanners are created equal..
The systems that identify and decode barcodes have advanced at a high pace and continues to improve to offer faster, smaller, cheaper and more robust readers.
While 1D laser-based scanners are still manufactured and deployed today, the most significant advances in the reading system came from the invention of 2D readers. A 2D reader offered significant evolution possibilities as a consequence of using an image sensor. It provides additional features not previously possible. This includes taking photographs or recording videos, and enables more advanced functions, like document scanning, OCR (orthogonal character recognition), object detection, and dimensioning, as just a few popular application examples.
Teledyne-e2v image sensors are unique in this market offering benefits over other 2D sensor options. This comes largely from the fact that they are specifically designed for barcode reading application as opposed to more general purpose, consumer or automotive applications. This translates to non-compromised, powerful solutions that provide all the ingredients that enables a market leading end barcode scanner reader.
Teledyne-e2v has recently developed a range of new small pixel, low noise, global shutter CMOS image sensors with unique dedicated application features that offer significant cost reduction and/or performance improvements in Automatic Data Collection Systems (ADCS) and Auto Identification (AI) market applications. Sensor unit cost is clearly a top criteria in this market/application, but other tangible cost reducing factors must also be considered at system level such as illumination/optics reduction.
2D reader systems require a very rapid frame capture to avoid blurring. This requires that the shortest possible integration time is used. Also, to get maximum depth of field (DOF) or scanning range, often a very small optical aperture (typically F/8 or smaller) lens is used. The very small amount of photons that enter the pixel of the image sensor, combined with the very short integration time means that barcode reading can be considered a low-light application (see Figure 5). A global shutter is also advantageous when the barcode target is moving.
The principal sensor parameters that contribute to end-scanner performance are therefore unique to the application. Figure 4 lists some of the main sensor/barcode scanning performance criteria, and shows the Snappy sensor family which we will use as an example of an application focused cmos image sensor.
When the heat is on..
A closer look at the difference between the various components that make up the different noise parameters between 25°C junction temperature, and what happens to them at >65°C yields many limiting factors and should be considered in the choice of sensor. Spatial row and column fixed pattern noise is a particularly important parameter for barcode reading. Considering that FPN looks like straight horizontal/vertical lines, they can be mistaken for a barcode, or add miss-information to real barcodes in the image.
The semiconductor process used for the Snappy sensors shows just a few electrons of dark signal at 25°C, but even at above 65°C, only 77 electrons/second is observed. This assists the built-in FPN (fixed pattern noise) cancellation algorithms of both row and column noise to achieve just a few percent of FPN even at high operating-temperature.
The total readout noise; which combines both temporal and spatial elements, is very low at typically 3 electrons and does not degrade beyond use at high-temperatures. Insufficient sensor performance at high temperature results in additional system cost from the only cure; since more light is needed.
Unique Mono/color pixel filter patterns can combine the low-light sensitivity of monochrome pixels with the advantages of producing object colour data.
Imagers can have color pixels which can add increased object/label recognition possibilities to provide additional security features to eliminate spoofing or in cases where the barcode itself can’t be read. Sensors used are predominantly monochrome though since color imagers are less sensitive due to the lower transmission properties of the organic color filter materials used and also represent a lower spatial resolution due to the need to combine red, green, and blue pixels to produce on ‘coloured’ pixel. An interesting innovation can be found on Teledyne-e2v’s Jade image sensor which uses a combination of both monochrome pixels with a color pixel once in every four monochrome pixels. The much needed spatial resolution and sensitivity to read barcodes is preserved in this way, while offering a lower resolution color image to be captured simultaneously.
Innovative embedded application specific features
Achieving fast (Snappy) barcode reading is not only a function of frame readout rate. While the maximum FPS could be a limiting factor, Snappy sensors do not compromise here by offering close to 120 frames per second in 8 bit operating mode. A unique power-up mode ensures that the device comes out of power up or standby with first acquired image (or FSE sub-image) at the datasheet SNR specification. This is not often the case for most automotive or other application oriented global shutter cmos sensors, which require several complete frames to be wasted before they stabilize and reach their reported data sheet SNR performance values. This unique first frame read ability after power up is also a differentiating factor for cameras that result simply in highest speed barcode reading or ‘snappyness’ of scanning to the end-user, and higher productivity levels for their organization.
Two most innovative patented features that Teledyne-e2v’s Professional Imaging teams have invented and featured in the latest Snappy family of sensors, designed to offer the fastest barcode scanning, identification and decoding in scanning end-products are highlighted below.
- Fast Self Exposure (FSE) mode, featured on Snappy 2MP and 5MP CMOS sensors:
FSE mode enables optimsation of the exposure time in varying light (see Figure 7). Bringing huge benefits compared with conventional auto exposure methods in terms of convergence time and robustness, the FSE mode is completely user programmable and provides a constant fast read benefit to end users, self-adapting to any kind of lighting or dynamic lighting environment, with almost no impact on the frame rate.
The patented FSE mode uses a number of on-chip components to enable this feature:
(a) A unique column ADC architecture which can allow 4 different integration periods to be set on consecutive columns, and then repeated across the array to produce 4 lower resolution images at different exposure values. It can also be used as a powerful HDR image capture feature .
(b) Incremental sub-sampling in the row direction of up to 1/64 lines.
(c) On chip statistics also count saturated pixels and provide in parallel a 16 bins histogram output, directly readable in the footer part of the image, for the currently active frame or region.
(d) An RoI mode to be applied to the FSE sub-frame. Multiple regions are possible also, as well as regions within a region.
(e) A fine control to use either the histogram value, an average value or a combination.
(f) Programmable registers to offer intuitive user-control and setup.
The end-application advantage again becomes scanning speed, since the FSE mode converges in typically much less than 10% of the frame period. Conventional embedded AEC controls in cmos sensors use asymptotic techniques to avoid flashing and provide a slow convergence to the target image contrast level. However, the fact that numerous frames are expended in the process, then this is too slow for the barcode reading application.
- Smart RoI mode, featured on the Snappy 5MP sensor:
Smart RoI uses an on-chip algorithm that detects a barcode or multiple barcodes in the current image. The barcode decoding image processing system requires that the area of the image containing a code is discerned from the rest of the image frame, in order to only analyse the area of interest. Whilst this operation is often performed on FPGA or CPU, the number of gates/RTC and processing horse-power devoted to this task is very high and adds significant cost and complexity constraints to the choice of processing engine.
Embedding this barcode detection task in the sensor brings cost reduction from reducing considerably the processing overhead, but also some performance/stability advantages by performing this inside the sensor compared to much further down the digital signal flow chain. The detected 1D or 2D barcodes in the active frame are presented as blocks with X/Y coordinates reported as readable metadata in the footer (i.e. non-visible) region of the image data. Many regions (or barcodes) can be detected simultaneously. Some other forms of repetative signatures such as printed character strings can also be detected for example in OCR applications. This feature also operates efficiently in applications where codes/objects/cameras are moving.
Significant cost savings and system simplification are the principal advantages. The use of 5MP sensors in barcode applications is limited mainly to high-end applications due to cost of larger optics and greater processing power needed to run real-time image processing/decoding on 5 million pixels, which offsets the principle advantage of offering greater scanning range or area. However, the small pixels of the Snappy sensor family and the reduction of processing overhead due to the on-chip Smart RoI feature, lower cost points at system level are achieved. Power consumption and the ability to recognize many barcode symbols in a single image frame are becoming drivers for higher resolution sensors, particularly in the e-commerce logistics market.
Both FSE and Smart RoI features can be run simultaneously in the Snappy 5MP sensor and ensures rapid, robust operation, even with constant changes in ambient light.
The Snappy family of sensors are designed to address real application issues and provide useful cost/power saving features to make processing circuits less complex, less power-hungry and less expensive. We have also demonstrated that the performance can be enhanced when treating data at source within the sensor itself. Other types of MV applications such as inspection, metrology, OCR, etc can benefit from the Snappy sensors performance and embedded features despite them being heavily focused on the barcode reading application/market. Some other areas of usage are in Embedded Vision Systems, IoT Edge Devices, Drones, Augmented Reality, Biometric Systems, etc.
Extending value by integrating optics and advanced mems based auto focus with Snappy sensors
Consumer camera optical modules are not options for the rugged long-term operation required in B2B industrial applications. Furthermore, the lens performance and optical characteristics are often very specific when decoding a barcode for example, where the working distance needs to be maximized, and optical aberrations such as optical MTF need to be minimized in order to efficiently decode barcodes with smaller and smaller feature sizes below Nyquist limits. Mipi Optical Modules (MOM) that combine the powerful Snappy image sensors with high performance optics increase system value and save on development time and cost.
Mipi Optical Modules are ideal solutions for embedded vision applications with some customization of the optics available to offer uncompromised performance and flexibility in a small 20mm x 20mm module. Teledyne-e2v is starting to sample end customers with the first 2MP Mipi Optical Module and will announce both a 5MP version, and a 2MP Auto Focus version that uses a robust and lightweight MEMS based element to perform the auto focus. The singular benefit of autofocus comes from enabling bigger effective optical apertures to be used compared with a fixed focus optical system to achieve same or improved scanning range or working distance, but requiring far less illumination power. Achieving better than two F stops of improvement, a future 2MP MEMS Mipi Optical Module will achieve additional performance and/or cost improvements in industrial imaging applications, barcode reading being one of the principal application segments that can make good use of this enhanced value. Product announcement is planned for mid-2020. In the meantime, a evaluation platform is available of a novel open-loop ‘multi-focus’ feature, that assures maximum working range with highest frame rates, and has the MEMS auto-focus element fitted to a conventional Snappy 2MP demokit.
New trends and changes caused by the e-commerce explosion
The huge double digit CAGR growth target (>25% per year) of the e-commerce boom is creating a shift in requirements in warehouse fulfilment centers, but also in the traditional ‘bricks and mortar’ retail point of sale market as a response, to retain and also grow their market share. Retail markets are undergoing major changes to improve the customer experience and minimize the delay through the checkouts by using un-manned automated ‘self-scanning’ systems. The success of these systems will depend on reliable barcode identification and decoding, but also on more sophisticated object recognition tasks that will see increased use of color imagers.
The requirements for higher speed or higher throughput of future scanners and cameras is a must-have if the high growth potential is to be fully realized. Higher resolution sensors with wider formats will allow for faster speeds in the future, and enable larger surface areas (containing many packages and barcodes in the image), to cover entire surface areas in warehouses in the future.
At sensor level, we will see the adoption of curved sensor technology with its advantage of reducing the complexity of external optics and alleviate some of the limitations with smaller pixels with regard to the diffraction limits of lens today. There are two benefits then; one that simplifies and reduces the cost of optics, and the other; which allows smaller pixels below 2.5µ to be employed without the MTF reduction impact of reaching optical diffraction limits.
The requirement to measure or dimension a package as well as scan its barcodes in modern warehouses is needed to meet the efficiency targets of emerging e-commerce fulfilment centers. Every square centimeter of valuable storage and transportation space must be used to its maximum. Monitoring the three-dimensional size of packages throughout the transport and logistics chain, in addition to reading associated codes/text labels using 2D scanning is currently performed by using two separate cameras; one for 3D (often using structured light or stereo vision based 3D techniques), in conjunction with a non-correlated 2D camera.
Efforts are now being made to invent a 2D plus 3D cmos sensor that will provide a conventional 2D image and 3D point cloud simultaneously. Teledyne-e2v is commited to the leading edge of innovation in these emerging market areas, and has future product roadmaps and IP aligned with the demands of next generation cameras and systems, that will continue to evolve into the foreseeable future, with focus on the specifics of their application.
