WO2007010473A2 - Automatic method and system for visual inspection of railway infrastructure - Google Patents

Automatic method and system for visual inspection of railway infrastructure Download PDF

Info

Publication number
WO2007010473A2
WO2007010473A2 PCT/IB2006/052438 IB2006052438W WO2007010473A2 WO 2007010473 A2 WO2007010473 A2 WO 2007010473A2 IB 2006052438 W IB2006052438 W IB 2006052438W WO 2007010473 A2 WO2007010473 A2 WO 2007010473A2
Authority
WO
WIPO (PCT)
Prior art keywords
sub
images
searching
type
phase
Prior art date
Application number
PCT/IB2006/052438
Other languages
French (fr)
Other versions
WO2007010473A3 (en
Inventor
Arcangelo Distante
Massimiliano Nitti
Ettore Stella
Pier Luigi Mazzeo
Francescomaria Marino
Original Assignee
Consiglio Nazionale Delle Ricerche
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Consiglio Nazionale Delle Ricerche filed Critical Consiglio Nazionale Delle Ricerche
Priority to US11/988,449 priority Critical patent/US20090196486A1/en
Priority to EP06780106A priority patent/EP1904972A2/en
Priority to JP2008522143A priority patent/JP2009501855A/en
Publication of WO2007010473A2 publication Critical patent/WO2007010473A2/en
Publication of WO2007010473A3 publication Critical patent/WO2007010473A3/en
Priority to US13/564,551 priority patent/US20130170709A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • G06T7/0004Industrial image inspection
    • G06T7/0008Industrial image inspection checking presence/absence
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B61RAILWAYS
    • B61LGUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L23/00Control, warning, or like safety means along the route or between vehicles or vehicle trains
    • B61L23/04Control, warning, or like safety means along the route or between vehicles or vehicle trains for monitoring the mechanical state of the route
    • B61L23/042Track changes detection
    • B61L23/047Track or rail movements
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B61RAILWAYS
    • B61LGUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L23/00Control, warning, or like safety means along the route or between vehicles or vehicle trains
    • B61L23/04Control, warning, or like safety means along the route or between vehicles or vehicle trains for monitoring the mechanical state of the route
    • B61L23/042Track changes detection
    • B61L23/048Road bed changes, e.g. road bed erosion

Definitions

  • the present invention relates to a visual inspection system for the maintenance of railway infrastructures.
  • the present invention relates to a system for automatically detecting the presence/absence of the coupling locks fastening the rails to the sleepers.
  • Such systems can increase the ability in recognizing the defects and in reducing the inspection time so as to guarantee more frequent maintenance works of the railway network.
  • a rail can be fastened to the sleepers by using two kinds of coupling locks: hexagonal-head bolts or hook bolts.
  • the first one has a regular hexagonal shape with a random orientation
  • the second one has a more complex hook-like shape which can be oriented in one single direction.
  • the object of the present invention is to solve the problems mentioned above, by providing an automatic method for visually inspecting an infrastructure, for detecting members characterizing the infrastructure itself as defined in the independent claim Nr. 1.
  • An additional object of the present invention is to provide a corresponding automatic automatized system for visually inspectioning an infrastructure, for detecting the members characterizing the infrastructure itself as defined in claim Nr. 15.
  • the present invention by overcoming the cited problems of the known art, involves numerous and evident advantages.
  • the inspection system according to the present invention is wholly automatic, it can operate in off-line mode, as well as in the preferred real time mode and it does not require a calibrating phase depending upon the type of member to be controlled.
  • the human operator intervenes only in the process of selecting the images of the locking members to be controlled.
  • the inspection method according to the present invention does not put any limit to the shape of the members to be controlled, being able to be adapted both to geometrical and irregular profiles.
  • figure 1 is a block diagram indicating the system according to the present invention.
  • figure 2 is a view of the video acquisition system
  • figure 3 is a more detailed block diagram of the system according to the present invention.
  • figure 4 shows three images of sub-image windows extracted from the acquired images
  • figure 5 is a visual display of the software programme implementing the method according to the present invention
  • figure 6 illustrates the geometry of a rail
  • figure 7 shows schematically a general decomposition level of a bi-dimensional wavelet transformation
  • figure 8 shows the result of applying two 2-D DWT levels to a sub-image
  • figure 9 shows a card used as prototype development platform
  • figure 10 shows a visual display extracted from the development system of the card of fig. 9;
  • figure 11 shows a representation of a kernel used in the calculation of the 2-D transforms
  • figure 12 shows a general line of a shift-register of the system hardware
  • figures 13A, 13B and 13C are graphs showing the presence detections of members, with respect to the analyzed lines;
  • figure 14 is a display extracted from the simulation report of the system according to the present invention.
  • figure 1 shows a block diagram of an inspection system 1 according to the present invention.
  • the system 1 comprises an acquisition unit of images 2.
  • the acquisition unit 2 is based upon a line camera DALSA PIRANHA 2 ® characterized by 1024 resolution pixels (maximum line rate of 67 kLine/s) and using the Cameralink ® protocol. Furthermore, it is provided with a PC-CAMLINK ® acquisition card (Imaging Technology CORECO ® ).
  • the acquisition unit 2 to the purpose of the here-described exemplifying application, is installed in the in the underneath part of a vehicle on rails, for example a railway car.
  • Figure 2 shows such installation.
  • the installation is such that, when the vehicle is moving, the camera can shoot the rails' subsequent images.
  • the acquisition unit 2 further comprises an apparatus of artificial lighting, preferably equipped with six small lights of OSRAM ® 41850 kind. In this way the image acquisition is strong with respect of the changes of natural lighting.
  • the camera is controlled by a "wheel encoder" which sends acquisition pulses at pre-established intervals.
  • the encoder sends a pulse every 3 mm rolling of the vehicle's wheels, thus determining a resolution along the direction of the rails (main motion direction) equal to 3 mm independently from the vehicle's speed.
  • the pixel resolution along the direction orthogonal to the motion is instead determined to 1 mm.
  • the acquired images are transmitted in real time to a processing unit 3 which provides for inspecting the image sequence in order to detect the presence/absence of characteristic members of the infrastructure.
  • a processing unit 3 which provides for inspecting the image sequence in order to detect the presence/absence of characteristic members of the infrastructure.
  • the processing modes implemented for detecting the presence (or absence) of the hexagonal-head bolts, fastening the rails to the sleepers will be described. Therefore, the processing unit 3 provides as result a detailed report about the status of the examined rails and in case showing the indications necessary to detect spacially the tracts wherein a problem has been identified such as, for example, the absence of a bolt.
  • FIG. 3 illustrates, through a functional block diagram, the processing modes implemented by the unit 3 in order to obtain the desired result.
  • the sequence of acquired images is supplied as input to a first processing sub-unit 4 which functionally provides, among other things, for the extraction of the geometrical coordinates of the rail under examination (RD&TB block).
  • the RD&TB block is advantageously implemented in hardware on a FPGA, thereof more details will be given hereinafter.
  • the RD&TB block serves to determine the track position in the acquired image.
  • the implemented technique is an accommodation of the method known under the name of EigenFaces.
  • the method is an extension of the Principal Component Analysis (PCA) and it is composed of two phases: a "data reduction” phase wherein the grey levels are mapped in a suitable space (Component Space) so as to reduce the data to be handled, and a "supervisioned classification” phase, based upon a neural network, to detect the track.
  • PCA Principal Component Analysis
  • Be ⁇ x,y a track image with M rows and N columns.
  • the image I is part of an input image.
  • Be r, (/-1..P, P ⁇ N) a set of P vectors composed by Q ⁇ N pixels extracted from the rows of the image I and selected among the data acquired so as to construct the matrix:
  • the vectors r,- are chosen so as to select track examples, acquired under different lighting conditions.
  • the matrix A has a size of N rows and P columns. From A, the covariance matrix can be constructed:
  • the matrix NxN, C contains information about the mutual relationships between the track images r, .
  • the eigenvalues ⁇ j of C are the variances of data for each one of the ii/.
  • the corresponding eigenvectors L (L ⁇ N) sufficient to represent 90% of the information content of the input data.
  • a general vector r' can be expressed as: r' « ⁇ ,u ; + ⁇ ' wherein ⁇ ' is the average of r'.
  • the track detection phase is based upon training a multilayer perceptron (MLP) on a training set containing examples r, of tracks and not tracks.
  • MLP multilayer perceptron
  • the test phase consists in scanning each input image line with a window with size Q (the track size). For each window position, the searching process extracts the vector r' and it evaluates by means of MLP if r' is or is not centered onto the track.
  • the rail width is 301 pixels
  • the detection system exploits a MLPNC constituted by three layers of neurons (input, hidden and output layer).
  • the biases and the weights are calculated by using the Error Back Propagation technique with adaptive learning rate and a training set with more than 800 samples.
  • the processing unit 3 further comprises a prediction sub-unit 5, identified in figure 3 with the acronym PAB. The same sequence of acquired images is provided as input to the sub-unit 5 PAB, together with the data related to the rail geometrical coordinates.
  • the sub-unit 5 PAB utilizes the data received as input to extract from the acquired images the sub-images candidate to contain the hexagonal-head bolts and only these windows will be the inspection subject ones.
  • sub-image windows of 24x100 pixel have been provided. Some examples of such windows are shown in figure 4.
  • a detection sub-unit 6 also designated with the acronym BDB (Bolt Detection Block) and which will be described in detail.
  • the sub-unit 5 PAB provides the position of the bolts along the direction orthogonal to the motion y.
  • a searching phase inside the sub-images is provided.
  • two searching modes are advantageously utilized: an exhaustive search and a jump-like search.
  • the searching areas at the (well-known) distance Dx from the rail position are examined, until the first occurrence of the left and right bolts are found at the same time (at the same y).
  • this position (position A) is determined and stored and the analysis continues in this way until the second occurrence of both bolts (position B) is identified. Therefore, the distance along the direction y between B and A is calculated, designated in figure 5 with Dy.
  • the searching process passes to the jump-like searching mode. In fact, as it is known, the distance along the direction y between two adjacent sleepers is fixed.
  • the jump-like search utilizes Dy to jump only into the areas candidate to contain the windows involved by the hexagonal-head bolts, by saving computation time and accelerating the performances inside the whole system.
  • the sub-unit 5 If, during the jump-like search, the sub-unit 5 does not find the bolts in the position wherein it is expected, it stores the error position (the one which has generated the alarm) in a log-file and it starts again with the exhaustive search.
  • a pseudo-code describing the switching from the exhaustive search to the jump-like search is shown hereinafter by way of example:
  • the output of the sub-unit 5 PAB is then provided as input to a sub-unit 6 for detecting the bolts (BDB).
  • the sub-unit BDB combines the result of two different processings. It comprises a pre-processing module 7 of the sub- images by means at least a transform function.
  • pre-processing module 7 comprises means 7', 7" for the simultaneously application of two transforms to the input data, in particular two discreet Wavelet transforms 2-D (DWT), which reduce significantly the input space size.
  • DWT discreet Wavelet transforms 2-D
  • the data outgoing from the pre-processing module, and specifically the results of the two transforms, are provided as input to a classification module 8.
  • the classification module comprises means 8' , 8" to carry out two different classifications, in particular two respective classifiers based upon neural networks (Multi Layer Perceptron Neural Classifiers, MLPNC). From tests carried out by means a prototype, BDB provides an accuracy of
  • the whole detection sub-unit 6 is preferably implemented in hardware based upon FPGA (able to carry out BDB processings in 8.09 ⁇ s), in cooperation with a prediction algorithm (which, by using the geometry of the railway infrastructures, extracts from the long video sequence a few windows to be analyzed) allows "real time" performances. For example, a long sequence of images covering 9 Km has been inspected at an average rate of 152 km/h, with peaks of 201 km/h.
  • the process, therewith the system according to the present invention classifies the bolts, is based upon classifiers based upon MLPNC neural networks.
  • the MLPNC computational performances are strictly linked to the prediction algorithm for identifying the search windows candidate to contain the image to be inspected and to the size of the inputs' space (that is, the number of coefficients describing the image).
  • the system calculates the distance between the subsequent hexagonal bolts and, based upon this information, predicts the position of the windows wherein the bolt presence is expected.
  • the system utilizes a feature of the extraction algorithm which allows preserving all important information about the input images in a small set of coefficients.
  • This algorithm is based upon a Wavelet discreet transform 2-D DWT, considering that DWT concentrates the significant variations of the input images in a reduced set of coefficients. In the specific case, both a compact wavelet introduced by Daubechies and a
  • Haar DWT also known as Haar transform
  • the input windows are reduced to two sets of 150 coefficients (that is, DJ-L 2 and HJ-L 2 ), resulting of the Daubechies DWT (DDWT) and of the Haar DWT (HDWT), respectively.
  • the two sets DJLL 2 e HJ-L 2 are then provided to the classification blocks Daubechies Classifier (DC) and Haar Classifier (HC), the outputs thereof are combined with a logic operation of AND to provide the output of the classification module 8 (MLPN).
  • the wavelet transform is notoriously a mathematic technique which decomposes a signal in the time domain by using expanded/contracted versions, translated by a single base function of finite duration, called wavelet prototype. This differs from the traditional transforms (for example, Fourier Transform, Cosine Transform, etc.), which utilize base functions of infinite duration.
  • the continuous mono-dimensional (1-D) wavelet transforms of a signal x(f) is:
  • the bi-dimensional (2-D) transforms DWT operate through a multi-level decomposition process.
  • a general 2-D DWT decomposition level j is shown in figure 7. It can be seen as an additional decomposition of a set 2-D of data LL ⁇ (LL 0 is the input original image) in four sub-bands LL j , LH 1 , HL 1 and HHj.
  • the capital letters and the position thereof relate to the application of monodimensional filters (L for low-pass filter, H for high-pass filter) and to the application direction (first letter for the horizontal direction, second letter for the vertical).
  • the band LL 1 is a rough approximation of LL j ⁇ .
  • N 1 N 0 Il
  • a compact wavelet orthonormal basis introduced by Daubechies is an excellent device to characterize the hexagonal- head bolts with a small number of features containing most part of useful information.
  • the sub-unit 5 provides 24x100-pixel sub-images so that they are examined. Different DWT were tested by varying the number of decomposition levels with the purpose of reducing this number without loosing accuracy. The best compromise was reached by the sub-band LL 2 consisting in only 6x25 coefficients.
  • the block calculates also the DWT sub-band LL 2 of Haar, since the intersection of the results of the two classifiers practically avoids completely the detection of false positives.
  • neural networks are utilized in several applications as “rule generation”, approximation of functions, routing, adaptive control, resource allocation, prediction and distribution of the workload, “collision avoidance”, “preference assessment”, etc.
  • the classifiers based upon neural networks have an undeniable advantage with respect to the techniques based upon geometric approaches since they do not require geometric models for representing the objects.
  • two MLPNC classifiers 8' and 8" are used, respectively trained for the DDWT (Daubechies) and the HDWT (Haar).
  • the two classifiers DC and HC have the same topology (they differ only in the weight value) and are constituted by three levels of neuron (input, hidden and output level).
  • H_n k H _w m>k H _n n (E14 1 )
  • D_w nhk and D_w k " fi [H _w m ' J , e H _ w k " fi ] are the weights between the first/second and the second/third level, respectively.
  • the activation function f(x) with a range ]0, 1 [, for both levels, is:
  • D_nl ⁇ H _nl ⁇ assumes values between 0 and 1 and it provides a measure of the reliability with respect to the presence of the object to be detected in the current search window, according to the DC [HC].
  • Presence (D_n TM > 0.9) AND (H _ ⁇ > 0.9) ( E17 ) so as to provide the final result of the Classifiers.
  • the biases and the weights are calculated by using the Error Back Propagation algorithm with an adaptative learning rate and a training set of more than 1 ,000 examples.
  • the FPGAs are the first devices which implement the state of art of the silicon technology.
  • the FPGAs have been created to develop simple logic, currently they represent the systems' heart in different field, such as precision measurement systems, multimedia, image processings, signal processing, medical instrumentation, cryptology/cryptoanalysis, power systems, video compression, communication systems, control systems, image recognition, database for recognizing finger-prints in real-time, etc.
  • the blocks with greater computational load have been implemented in hardware: in particular the sub-unit 6 BDB, in the pre-processing module DWTPB and classification module MLPNCB.
  • the pre-processing module DWTPB in the pre-processing module DWTPB and classification module MLPNCB.
  • MLPNCB classification module
  • FIG. 10 shows a display of Quartus IITM CAD tool with a highest schematic view.
  • the architecture can be interpreted as memory: the operation starts when the host "writes" a 24x100-pixel window to be analyzed. In this phase, the host recalls the double-gate memories inside the INPUTJNTERFACE (pin address[9..O]) and sends the 2400 byte to the data input pins[63..O] under the form of 300 64-bit words. - As soon as the machine has completed its work, the output line irq signalls that the results are ready. At this point, the host "reads" them by recalling the FIFO memory inside the OUTPUTJNTERFACE.
  • the INPUTJNTERFACE block receives the input data and separates the input phase from the processing phase, this mainly to make the processing phase synchronous and independent from the delays which could be onto the bus PCI during the input. Furthermore, it allows operating at a higher frequency (clkHW signal) with respect to the I/O (clkPCI signal).
  • the Daubechies 2-D DWT pre-processing is carried out by means of the cooperation of the SHIFTREGISTERS block with the DAUBJ_L2_FILTER one.
  • the processing in floating point has been discarded and the calculation in fixed point precision has been adopted.
  • the attention has been focused on the latter.
  • each step of the splittable method provides the results with a different dynamics, in this way the hardware used at a determined step results to be unusable to carry out additional steps.
  • DAUB_LL2_FILTER produces the coefficients LL 2 , after a latency of 11 ccs, at a rate of 1 coefficient/cc. These are now expressed in 35 bit, due to the increase in dynamics, and they are input into 1 LEV_MLPN_CLASSIFIER through D_LL2[34..O ].
  • H LL 1 (UJ) ⁇ z ⁇ pW + KAj + 1) (E20)
  • the same SHIFTREGISTERS block is utilized, used for Daubechies DWT, and a HAAR-LL 2- FILTER block.
  • the HAAR-LL 2- FILTER block sums the data which coming from OUT-TO-HAAR-16bytes[255..O] which represent the values of the pixels p(m , n) of the 4x4 window centred onto the scrolling 16x16 matrix, implemented by the SHIFTREGISTERS.
  • HAAR_LL2_FILTER produces a coefficient (expressed by 12 bits) per block cycle and it sends it to the 1LEV_MLPN_CLASSIFIER block through H_LL2[11..O].
  • Equation (E23) and (E23 1 ) represent the arguments of the activation function of (E21) and (E21 1 ). By doing this, such arguments are calculated in hardware and are sent to the host, which in software calculates f(p_x k ) , f ⁇ H_x k ) , (E22), and (E22 1 ). However, (E23) and (E23 1 ) represent 3,000 multiplications and 3,000 sums which are calculated in hardware, against 20 multiplications, 20 sums, 22 activation functions and the comparison to the threshold (E 17), calculated in the software by the host.
  • MACs Multiplier-and-ACcurnulators
  • the weights D__w m ' ⁇ and H_w m k have been previously stored in 20 LUTs during the setup (a LUT for each multiplier, which stores the 150 weights).
  • the accumulator of each D_MAC fc [ H_MAC ⁇ ] is initialized with D_biask [ H_hiask ] and it accumulates the products as soon as they are calculated by the multipliers.
  • the operation of the 1LEV_MLPN_CLASSIFIER block ends up after 5 ccs from the last coefficients, D_LL 2 (5, 24) and H_LL 2 (5, 24), provided by DAUB_LL2_FILTER and by HAAR_LL2_FILTER.
  • D_LL 2 5, 24
  • H_LL 2 5, 24
  • DAUB_LL2_FILTER HAAR_LL2_FILTER.
  • OUTPUTJNTERFACE block. Furthermore, OUTPUTJNTERFACE serializes them by using a FIFO and it generates the irq signal which indicates that the results are ready to be collected. To end up, the host requires these results (read signal) and receives them on the dataread[63..O] (1 word/cc).
  • Table 1 sums-up the resource engagement required by the architecture.
  • the Table shows also a comparison between the resources available on the StratixTM EP1S60F1020C6 FPGA. In the partial use of the resources it is to be considered that also the previously described RD&TB block must be implemented on the same FPGA.
  • the image set adopted for training the neural network of the classifier includes 391 positive examples of hexagonal bolts with different orientation, and 703 negative examples represented by 24x100-pixel windows exctracted from the acquired video sequence.
  • HC classifier is a neural classifier which operates with the sub-band LL 2 produced by the Haar DWT, and it has been designed and trained. HC reaches the same detection rate of DC, even if it detects several more false positives.
  • Haar DWT 22 (100%) 90 409% 200.0% 00
  • the report (file log) obtained by the above-mentioned test has been used as comparison term for the reports of similar experiments, which aim at defining the number of bits of the words to be used in the hardware design.
  • the software prototype has been modified by floating point in fixed point.
  • the different versions of the software procedures have been compiled with different precisions (that is number of used bits) both for the coefficients of the Daubechies filter and for the weights both of DC and HC.
  • the setting with 23 bits for the filter coefficients and with 25 bits for the weights of both classifiers has produced a lower accuracy, in detecting the visible bolts, of only 0.3% than the one obtained using the precision in floating point. This compromise has been considered acceptable and the hardware has been developed by using these specifications.
  • the present invention has been sofar described according to a preferred embodiment thereof, shown by only way of example and not with limitative purpose. It is to be meant that other embodiments may be provided, all to be considered belonging to the protection scope of the same, as defined by the enclosed claims.

Abstract

The present invention relates to a visual inspection system and method for the maintenance of infrastructures, in particular railway infrastructures. It is a system able to operate in real time, wholly automatically, for the automatic detection of the presence/absence of characterizing members of the infrastructure itself, for example the coupling locks fastening the rails to the sleepers.

Description

"AUTOMATIC METHOD AND SYSTEM FOR INFRASTRUCTURE VISUAL INSPECTION"
DESCRIPTION
The present invention relates to a visual inspection system for the maintenance of railway infrastructures. In particular, the present invention relates to a system for automatically detecting the presence/absence of the coupling locks fastening the rails to the sleepers.
The maintenance of the railway infrastructures is included in a particular applicative context therefor a periodical inspection is necessary to prevent dangerous situations. Usually, this operation is visually performed by specialized personnel which, periodically, runs along the railway network looking for the anomalies at sight.
Evidently, such manual inspection is slow, laborious and potentially risky as the provided results are strictly linked to the observer's ability in detecting possible anomalies and in recognizing critical situations.
With the growth of the high speed railway traffic, the companies of all over the world are interested in developing automatic inspection systems able to recognize the track's defects, the sleepers' anomalies, as well as the absence of coupling locks.
Such systems can increase the ability in recognizing the defects and in reducing the inspection time so as to guarantee more frequent maintenance works of the railway network.
As it is known, a rail can be fastened to the sleepers by using two kinds of coupling locks: hexagonal-head bolts or hook bolts.
These two kinds of locks mainly differ in the shape: the first one has a regular hexagonal shape with a random orientation, the second one has a more complex hook-like shape which can be oriented in one single direction.
The present invention will be described by referring to the case of octagonal- head bolts, even if, as it will be evident hereinafter, the same invention could find direct application in the other case as well as in the need for detecting other types of defects. Devices are well known in literature relating to the problems connected to the railway infrastructures, like for example a system for measuring the profile of the rotating track such as the one described in C. Alippi, E. Casagrande, F. Scotti, and V. Piuri, "Composite Real-Time Image Processing for Railways Track Profile Measurement," IEEE Trans. Instrumentation and Measurement, vol. 49, Nr. 3, pages 559-564, June 2000, or a system for detecting obstructions as described in K Sato, H. Arai, T. Shimuzu, and M. Takada, Obstruction Detector Using Ultrasonic Sensors for Upgrading the Safety of a Level Crossing," Proceedings of the IEE International Conference on Developments in Mass Transit Systems, pages 190-195, April 1998, a system recognizing the track's defects as described in Cybernetix Group (France), "IVOIRE: a system for rail inspection," internal documentation, http://www.cybernetix.fr and/or in Benntec Systemtechnik Gmbh, "RAILCHECK: image processing for rail analysis," internal documentation, http://www.benntec.com, a system for inspecting the status of the switches as described in A. Rubaai, "A neural-net-based device for monitoring Amtrak railroad track system," IEEE Transactions on Industry Applications, vol. 39, Nr. 2 , pages 374-381 , March-April 2003, etc.
Nevertheless, currently there are no systems working onto specific problems for recognizing the coupling members. The only existing methods are visual trade systems which take into consideration exclusively coupling members having regular geometrical profiles (such as the hexagonal bolts) and they use geometrical methods for recognizing images for analyzing the problem.
Nevertheless, such systems are very interactive and, in order to obtain the best performance, they need a human operator for calibrating each threshold. Therefore, when a different coupling member has to be examined, the calibrating phase must be performed again.
Therefore, the object of the present invention is to solve the problems mentioned above, by providing an automatic method for visually inspecting an infrastructure, for detecting members characterizing the infrastructure itself as defined in the independent claim Nr. 1.
An additional object of the present invention is to provide a corresponding automatic automatized system for visually inspectioning an infrastructure, for detecting the members characterizing the infrastructure itself as defined in claim Nr. 15.
Secondary features of the present invention are defined in the respective depending claims.
The present invention, by overcoming the cited problems of the known art, involves numerous and evident advantages. The inspection system according to the present invention is wholly automatic, it can operate in off-line mode, as well as in the preferred real time mode and it does not require a calibrating phase depending upon the type of member to be controlled. The human operator intervenes only in the process of selecting the images of the locking members to be controlled.
Furthermore, the inspection method according to the present invention does not put any limit to the shape of the members to be controlled, being able to be adapted both to geometrical and irregular profiles.
Additional advantages, as well as features and application modes of the present invention will be evident from the following detailed description of a preferred embodiment thereof, shown by way of example and not for limitative purposes, by referring to the figures of the enclosed drawings, wherein:
• figure 1 is a block diagram indicating the system according to the present invention;
• figure 2 is a view of the video acquisition system;
• figure 3 is a more detailed block diagram of the system according to the present invention;
• figure 4 shows three images of sub-image windows extracted from the acquired images;
• figure 5 is a visual display of the software programme implementing the method according to the present invention; • figure 6 illustrates the geometry of a rail;
• figure 7 shows schematically a general decomposition level of a bi-dimensional wavelet transformation;
• figure 8 shows the result of applying two 2-D DWT levels to a sub-image;
• figure 9 shows a card used as prototype development platform; • figure 10 shows a visual display extracted from the development system of the card of fig. 9;
• figure 11 shows a representation of a kernel used in the calculation of the 2-D transforms;
• figure 12 shows a general line of a shift-register of the system hardware; • figures 13A, 13B and 13C are graphs showing the presence detections of members, with respect to the analyzed lines;
• figure 14 is a display extracted from the simulation report of the system according to the present invention; and
• figures 15A and 15B are graphs showing the switching between two searching modes.
Hereinafter in the description the figures mentioned above will be referred to. By referring to figure 1, it shows a block diagram of an inspection system 1 according to the present invention.
The system 1 comprises an acquisition unit of images 2.
The acquisition unit 2 is based upon a line camera DALSA PIRANHA 2® characterized by 1024 resolution pixels (maximum line rate of 67 kLine/s) and using the Cameralink® protocol. Furthermore, it is provided with a PC-CAMLINK® acquisition card (Imaging Technology CORECO®).
The acquisition unit 2, to the purpose of the here-described exemplifying application, is installed in the in the underneath part of a vehicle on rails, for example a railway car. Figure 2 shows such installation.
The installation is such that, when the vehicle is moving, the camera can shoot the rails' subsequent images.
In order to reduce the effects of the variable conditions of the natural brightness, the acquisition unit 2 further comprises an apparatus of artificial lighting, preferably equipped with six small lights of OSRAM® 41850 kind. In this way the image acquisition is strong with respect of the changes of natural lighting.
Furthermore, in order to synchronize the data acquisition, the camera is controlled by a "wheel encoder" which sends acquisition pulses at pre-established intervals. Preferably the encoder sends a pulse every 3 mm rolling of the vehicle's wheels, thus determining a resolution along the direction of the rails (main motion direction) equal to 3 mm independently from the vehicle's speed. The pixel resolution along the direction orthogonal to the motion is instead determined to 1 mm.
The acquired images are transmitted in real time to a processing unit 3 which provides for inspecting the image sequence in order to detect the presence/absence of characteristic members of the infrastructure. In particular, hereinafter the processing modes implemented for detecting the presence (or absence) of the hexagonal-head bolts, fastening the rails to the sleepers, will be described. Therefore, the processing unit 3 provides as result a detailed report about the status of the examined rails and in case showing the indications necessary to detect spacially the tracts wherein a problem has been identified such as, for example, the absence of a bolt.
This report could be then advantageously exploited to provide for the maintenance of the tracts by accessing in direct manner to the tracts having defects, without the need of any additional inspection. The subsequent figure 3 illustrates, through a functional block diagram, the processing modes implemented by the unit 3 in order to obtain the desired result.
In particular, the sequence of acquired images is supplied as input to a first processing sub-unit 4 which functionally provides, among other things, for the extraction of the geometrical coordinates of the rail under examination (RD&TB block).
Considering the high computational complexity, in order to obtain the wished performances, the RD&TB block is advantageously implemented in hardware on a FPGA, thereof more details will be given hereinafter. The RD&TB block serves to determine the track position in the acquired image.
The implemented technique is an accommodation of the method known under the name of EigenFaces. The method is an extension of the Principal Component Analysis (PCA) and it is composed of two phases: a "data reduction" phase wherein the grey levels are mapped in a suitable space (Component Space) so as to reduce the data to be handled, and a "supervisioned classification" phase, based upon a neural network, to detect the track.
Be \{x,y) a track image with M rows and N columns. The image I is part of an input image. Be r, (/-1..P, P≥N) a set of P vectors composed by Q<N pixels extracted from the rows of the image I and selected among the data acquired so as to construct the matrix:
A=[M hP] (E1) wherein: hi = ri - μi (E2) being: μi = [μi,.,μ i]T (E3)
Wherein the symbol//,- denotes the intensity average in r,-.
The vectors r,- are chosen so as to select track examples, acquired under different lighting conditions.
The matrix A has a size of N rows and P columns. From A, the covariance matrix can be constructed:
C=AAT (E4)
The matrix NxN, C contains information about the mutual relationships between the track images r, . According the PCA analysis, the eigenvectors u;- (/=1..N) of C define a new reference space wherein the variance between the data is maximized. The eigenvalues λj of C are the variances of data for each one of the ii/.
Therefore, the eigenvectors uy ordinated so as: λk > λk+1 (k=1 , .., N-1) (E5) mean that the set of input data projections onto uk has a greater variance than those related to U^+1. Therefore, the eigenvalues λ,- induce an order relationship onto the components uy.
By setting thresholds to the eigenvalues λ,- it is possible to select the corresponding eigenvectors L (L<N) sufficient to represent 90% of the information content of the input data. Be λ/ (/=1..L, L<N) the selected components, a general vector r' can be expressed as: r' «∑α,u; +μ' wherein μ' is the average of r'. From a computational point of view the eigenvectors and the eigenvalues of C can be estimated by the Single Value Decomposition (SVD) of the matrix A wherein the coefficients a,- are evaluated by the scalar product: al = (r'-μ')ulT (E7)
In this scenario, the vector a'=[a1 aL]T (E8) can be considered a feature containing great part of information of r'. The track detection phase is based upon training a multilayer perceptron (MLP) on a training set containing examples r, of tracks and not tracks.
The test phase consists in scanning each input image line with a window with size Q (the track size). For each window position, the searching process extracts the vector r' and it evaluates by means of MLP if r' is or is not centered onto the track. The acquisition setup is constituted, as it has been said, by a camera which digitalizes lines of 1024 pixels, in order to detect the track centre, the edge pixels are eliminated and lines of only 800 pixels (Λ/=800) are considered. As the rail width is 301 pixels, the vector r,- with 401 pixels (50 edge pixels + 301 track pixels +50 edge pixels; Q=401) has been constructed. The matrixes A and C are derived according to (E1) and (E4), by using 450 examples of vectors r,- (P=450). Therefore, the eigenvectors U/and the eigenvalues λj are determined and it is experimentally checked that 12 eigenvectors are sufficient to represent 91% of the information content of the input data (L=12).
The detection system exploits a MLPNC constituted by three layers of neurons (input, hidden and output layer).
The input layer is formed by 12 neurons n1im (m=0..11) corresponding to the coefficients am+1 calculated by the (E7) onto the vector r' which must be classified. The hidden layer consists of 8 neurons nz,k {k=0..7); they derive from the propagation of the first layer according to: f ii Λ nU = / biaSl,k + Σ Wl,m,knlm (FQ)
V m=0 / V ' whereas the single neuron n3i0 of the output layer is given by:
«3,o = / bias2fi +∑w2iM«u wherein Wi,m,/c and w2,/c,o are the weights between first/second and second/third layers. The activation function f(x), which has codomain ]0, 1 [, for both layers is:
Figure imgf000008_0001
In this scenario, the output of the classifier n3,o which is in the range ]0,1[ gives a confidence measure about how much the vector r' is centred onto the track.
The biases and the weights are calculated by using the Error Back Propagation technique with adaptive learning rate and a training set with more than 800 samples. The processing unit 3 further comprises a prediction sub-unit 5, identified in figure 3 with the acronym PAB. The same sequence of acquired images is provided as input to the sub-unit 5 PAB, together with the data related to the rail geometrical coordinates.
The sub-unit 5 PAB utilizes the data received as input to extract from the acquired images the sub-images candidate to contain the hexagonal-head bolts and only these windows will be the inspection subject ones.
In the described case, sub-image windows of 24x100 pixel have been provided. Some examples of such windows are shown in figure 4.
These windows are then provided as input to a detection sub-unit 6, also designated with the acronym BDB (Bolt Detection Block) and which will be described in detail.
By referring to figure 5, due to the rail structure, the distance Dx between rail and coupling locks is constant and known in advance. In this way, the automatic detection of the rail and the tracking of the same results to be fundamental in determining the position of the bolts along the moving direction x. The detection procedure is carried out by the RD&TB block.
The sub-unit 5 PAB provides the position of the bolts along the direction orthogonal to the motion y. To achieve this purpose a searching phase inside the sub-images is provided. For such searching phase two searching modes are advantageously utilized: an exhaustive search and a jump-like search.
In the first (exhaustive) search kind the searching areas at the (well-known) distance Dx from the rail position are examined, until the first occurrence of the left and right bolts are found at the same time (at the same y). At this point, this position (position A) is determined and stored and the analysis continues in this way until the second occurrence of both bolts (position B) is identified. Therefore, the distance along the direction y between B and A is calculated, designated in figure 5 with Dy. At this point, the searching process passes to the jump-like searching mode. In fact, as it is known, the distance along the direction y between two adjacent sleepers is fixed. Furthermore, the jump-like search utilizes Dy to jump only into the areas candidate to contain the windows involved by the hexagonal-head bolts, by saving computation time and accelerating the performances inside the whole system.
If, during the jump-like search, the sub-unit 5 does not find the bolts in the position wherein it is expected, it stores the error position (the one which has generated the alarm) in a log-file and it starts again with the exhaustive search. A pseudo-code describing the switching from the exhaustive search to the jump-like search is shown hereinafter by way of example:
Do Start image sequence to End image sequence; Repeat
Exhaustive search; If the first left and right bolt is found it stores this position (A); until the second left and right bolt is found; it stores this position (B); determines the distance along y between B and A; Repeat jump-like search;
Until the bolts are found where expected; end do
As already shown, the output of the sub-unit 5 PAB is then provided as input to a sub-unit 6 for detecting the bolts (BDB). In order to avoid detecting false positives, the sub-unit BDB combines the result of two different processings. It comprises a pre-processing module 7 of the sub- images by means at least a transform function. In particular, such pre-processing module 7 comprises means 7', 7" for the simultaneously application of two transforms to the input data, in particular two discreet Wavelet transforms 2-D (DWT), which reduce significantly the input space size. The data outgoing from the pre-processing module, and specifically the results of the two transforms, are provided as input to a classification module 8. The classification module comprises means 8' , 8" to carry out two different classifications, in particular two respective classifiers based upon neural networks (Multi Layer Perceptron Neural Classifiers, MLPNC). From tests carried out by means a prototype, BDB provides an accuracy of
99.6% in detecting the existing bolts and of 95% in detecting the absent bolts, furthermore, utilizing the strategy of the results' intersection, it reveals only 1 false positive every 2,250 processed lines of video sequence.
Given the high computational complexity, the whole detection sub-unit 6 is preferably implemented in hardware based upon FPGA (able to carry out BDB processings in 8.09 μs), in cooperation with a prediction algorithm (which, by using the geometry of the railway infrastructures, extracts from the long video sequence a few windows to be analyzed) allows "real time" performances. For example, a long sequence of images covering 9 Km has been inspected at an average rate of 152 km/h, with peaks of 201 km/h.
The process, therewith the system according to the present invention classifies the bolts, is based upon classifiers based upon MLPNC neural networks. The MLPNC computational performances are strictly linked to the prediction algorithm for identifying the search windows candidate to contain the image to be inspected and to the size of the inputs' space (that is, the number of coefficients describing the image).
The system, by means of the prediction sub-unit 5, calculates the distance between the subsequent hexagonal bolts and, based upon this information, predicts the position of the windows wherein the bolt presence is expected. In order to reduce the size of the inputs' space, the system utilizes a feature of the extraction algorithm which allows preserving all important information about the input images in a small set of coefficients. This algorithm is based upon a Wavelet discreet transform 2-D DWT, considering that DWT concentrates the significant variations of the input images in a reduced set of coefficients. In the specific case, both a compact wavelet introduced by Daubechies and a
Haar DWT (also known as Haar transform) are preferably used in parallel.
By operating this pre-processing process the input windows are reduced to two sets of 150 coefficients (that is, DJ-L2 and HJ-L2), resulting of the Daubechies DWT (DDWT) and of the Haar DWT (HDWT), respectively. The two sets DJLL2 e HJ-L2 are then provided to the classification blocks Daubechies Classifier (DC) and Haar Classifier (HC), the outputs thereof are combined with a logic operation of AND to provide the output of the classification module 8 (MLPN).
This detects the presence/absence of bolts and provides a corresponding signal of Pass/Alarm which is displayed in real time (see figure 5). In case of alarm (that is absence of bolts), the spacial coordinates and other essential data are stored in a report file.
The function of logical AND avoids almost completely the detection of false positives.
In order to better understand the implemented processing modes, hereinafter some information related to the mathematics of the wavelet transforms and also of the neural classifiers are reported.
The wavelet transform is notoriously a mathematic technique which decomposes a signal in the time domain by using expanded/contracted versions, translated by a single base function of finite duration, called wavelet prototype. This differs from the traditional transforms (for example, Fourier Transform, Cosine Transform, etc.), which utilize base functions of infinite duration. The continuous mono-dimensional (1-D) wavelet transforms of a signal x(f) is:
Figure imgf000011_0001
wherein is the conjugated complex of the wavelet prototype, ; a
Figure imgf000011_0002
Figure imgf000011_0003
represents an expansion in time and b a translation in time. Due to the discreet nature (both in time and in space) of several applications, different Wavelet transforms (DWT) have been proposed based upon the signal nature, upon the time and spacial parameters.
The bi-dimensional (2-D) transforms DWT operate through a multi-level decomposition process. A general 2-D DWT decomposition level j is shown in figure 7. It can be seen as an additional decomposition of a set 2-D of data LL^ (LL0 is the input original image) in four sub-bands LLj, LH1, HL1 and HHj. The capital letters and the position thereof relate to the application of monodimensional filters (L for low-pass filter, H for high-pass filter) and to the application direction (first letter for the horizontal direction, second letter for the vertical). The band LL1 is a rough approximation of LL. The bands LH1 and HL1, respectively, store the changes along the horizontal and vertical directions of LLM, whereas HH1 shows the high-frequency components. As for each level along both directions it is necessary to decimate, each sub-band at level j is composed by NJXMJ elements, wherein N1=N0Il and Mj=M0/2. Different properties of the DWT can be emphasized by the use of different filters L and H. Thanks to this flexibility, the DWT was successfully applied to a wide range of applications: discrimination and segmentation of tissues, fractal analysis, image compression, recognition of radar objects, numeric analyses, edge extrapolation, biomedicine, etc. Furthermore, it was observed that a compact wavelet orthonormal basis introduced by Daubechies is an excellent device to characterize the hexagonal- head bolts with a small number of features containing most part of useful information.
Following the setup of the acquisition process of the system according to the present invention, the sub-unit 5 provides 24x100-pixel sub-images so that they are examined. Different DWT were tested by varying the number of decomposition levels with the purpose of reducing this number without loosing accuracy. The best compromise was reached by the sub-band LL2 consisting in only 6x25 coefficients.
By using the classifier described in the following Section, it guarantees an accuracy of 99.9% in recognizing the bolts from the input images.
At the same time, the block calculates also the DWT sub-band LL2 of Haar, since the intersection of the results of the two classifiers practically avoids completely the detection of false positives.
The neural networks are utilized in several applications as "rule generation", approximation of functions, routing, adaptive control, resource allocation, prediction and distribution of the workload, "collision avoidance", "preference assessment", etc.
Furthermore, their popularity is due to the image processing applications, especially in extracting information from the images. In fact, the classifiers based upon neural networks have an undeniable advantage with respect to the techniques based upon geometric approaches since they do not require geometric models for representing the objects.
According to the present invention two MLPNC classifiers 8' and 8" (DC and HC in figure 3) are used, respectively trained for the DDWT (Daubechies) and the HDWT (Haar). The two classifiers DC and HC have the same topology (they differ only in the weight value) and are constituted by three levels of neuron (input, hidden and output level).
The DC [and HC] input level is composed by 150 neurons D _nm [H _nm j (m =
0..149) corresponding to coefficients D_LL2(i, j) [H_LL2(i, j)] of the sub-band DJ-L2 [H_LL2] according to:
D_nm = DJUL1(Tn/ 25, mmod25) (E13) H _nm = H_LL2(m/ 25, mmod25) (E13!)
The DC [and HC] hidden level consists in 10 neurons D_nk" [H _nk\ (Zf=O..9); they derive from the propagation of the first level according to:
149
D_nk " = f\ D_biask + ∑D_wm>kD_nm (E14)
H_nk = H _wm>kH _nn (E141)
Figure imgf000013_0001
at last, the unique neuron D_nl \H _riQ ' J of the output level is given by:
D_nl + ∑D_wktQD_nλ (E15)
Figure imgf000013_0002
H _n0 " (E151)
Figure imgf000013_0003
wherein D_wnhk and D_wk"fi [H _wm' J, e H _ wk " fi ] are the weights between the first/second and the second/third level, respectively. The activation function f(x) , with a range ]0, 1 [, for both levels, is:
Figure imgf000013_0004
In this scenario, D_nl \H _nl\ assumes values between 0 and 1 and it provides a measure of the reliability with respect to the presence of the object to be detected in the current search window, according to the DC [HC].
D_rζ and H _n^' combine as follows:
Presence = (D_n > 0.9) AND (H _τζ > 0.9) (E17) so as to provide the final result of the Classifiers. The biases and the weights are calculated by using the Error Back Propagation algorithm with an adaptative learning rate and a training set of more than 1 ,000 examples.
As already indicated previously, some units and/or sub-units are implemented in hardware. Nowadays, the programmable logics play a strategic role in several fields. In fact, during the last twenty years, flexibility requirements have been strongly required so as to reduce more and more the time-to-market times. Furthermore, generally the FPGAs are the first devices which implement the state of art of the silicon technology.
Therefore, even if initially the FPGAs have been created to develop simple logic, currently they represent the systems' heart in different field, such as precision measurement systems, multimedia, image processings, signal processing, medical instrumentation, cryptology/cryptoanalysis, power systems, video compression, communication systems, control systems, image recognition, database for recognizing finger-prints in real-time, etc.
In order to allow the system according to the present invention to obtain real time performances, the blocks with greater computational load have been implemented in hardware: in particular the sub-unit 6 BDB, in the pre-processing module DWTPB and classification module MLPNCB. As prototype development platform a kit of Altera, PCI High-Speed
Development Kit Stratix™ Professional Edition (figure 9) has been adopted, which among other features has: a device Stratix™ EP1S60F1020C6 FPGA, 256-MByte PC333 DDR SDRAM,32-bit or 64-bit PCI and 8/16-bit, differential I/O up to 800 Mbps. The device Stratix™ EP1S60F1020C6 FPGA is equipped with 57,120 Look Up
Table (LUT), 18 DSP block and several memory elements of different sizes which globally reach 5,215,104 bits with a whole band of more than 10Tbits/s.
The environment for the design, simulation and test is the Quartusll™. Figure 10 shows a display of Quartus II™ CAD tool with a highest schematic view. The architecture can be interpreted as memory: the operation starts when the host "writes" a 24x100-pixel window to be analyzed. In this phase, the host recalls the double-gate memories inside the INPUTJNTERFACE (pin address[9..O]) and sends the 2400 byte to the data input pins[63..O] under the form of 300 64-bit words. - As soon as the machine has completed its work, the output line irq signalls that the results are ready. At this point, the host "reads" them by recalling the FIFO memory inside the OUTPUTJNTERFACE.
The INPUTJNTERFACE block receives the input data and separates the input phase from the processing phase, this mainly to make the processing phase synchronous and independent from the delays which could be onto the bus PCI during the input. Furthermore, it allows operating at a higher frequency (clkHW signal) with respect to the I/O (clkPCI signal).
The Daubechies 2-D DWT pre-processing is carried out by means of the cooperation of the SHIFTREGISTERS block with the DAUBJ_L2_FILTER one. In order to safeguard the resources and the calculation time, the processing in floating point has been discarded and the calculation in fixed point precision has been adopted. Furthermore, as exclusively the sub-band LL2 is of interest, the attention has been focused on the latter.
It can be noted that the 2-D DWT filter proposed by Daubechies has the 1-D L filter:
0,035226| -0,08544| -0,1350i l 0,45988[ 0,80689| 0,33267| (E18) And that the sub-band LL2 can be calculated in one single bidimensional step (instead of in the classic mode which provides two monodimensional steps shown in figure 7), followed by a decimation of 4 both along the rows and the columns. Figure 11 shows the symmetrical kernel 16x16 which has been applied. The splittable method is greatly efficient in computing all four sub-bands for each level. But the classification process of the system according to the present invention does not need additional sub-bands apart from LL2.
Furthermore, when the fixed precision is used, each step of the splittable method provides the results with a different dynamics, in this way the hardware used at a determined step results to be unusable to carry out additional steps.
The error (due to the fixed point precision) generated in a single step does not propagate and it can be easily controlled. On the contrary, when the splittable approach is used for calculating the sub-band LL2, the error propagates along the four steps. In this optics, the SHIFTREGISTERS implements a 16x16 matrix which scrolls, onto the 24x100 input window, by moving by 4 along the columns at each clock cycle. This is implemented by a route as indicated in figure 12, wherein the row/7 G=O..15) of the 16x16 matrix is represented. The shifting by 4 along the rows is implemented by the INPUTJNTERFACE which inserts in the y* row of the matrix only the pixels p(m, n) of the 24x100 (m=0..23, /7=0..99) input window, wherein (/ mod 4)=(m mod 4).
Every clock stroke sixteen contiguous rows of the input window in parallel in the SHIFTREGISTERS are sent at a rate of 64 bytes/cc (4 bytes for each line for 16 lines) through IN[511..O]. Simultaneously, all 256 bytes stored in the 16x16 memory are sent in parallel in the DAUB_LL2_FILTER through OUT_256bytes[2047..0].
DAUB_LL2_FILTER exploits the kernel symmetry (see figure 11), by adding the pixels coming from the cells (j, 0 to those from the cells (/, j) (/=0..15, /=0..15); subsequently, it calculates the products of there sums and of the diagonal elements of the matrix with the related filter coefficients and, to end up, it accumulates these products.
Consequently, DAUB_LL2_FILTER produces the coefficients LL2, after a latency of 11 ccs, at a rate of 1 coefficient/cc. These are now expressed in 35 bit, due to the increase in dynamics, and they are input into 1 LEV_MLPN_CLASSIFIER through D_LL2[34..O ].
The Haar transform is a very simple DWT, in fact the related 1-D filters are: L=[1/2,1/2] and H=[1/2, -1/2 ]. Consequently, any coefficient H_LL2(i, j) can be calculated in one single step:
H LL1(UJ) = γz∑∑pW + KAj + 1) (E20) In order to calculate such coefficient, the same SHIFTREGISTERS block is utilized, used for Daubechies DWT, and a HAAR-LL2-FILTER block. The HAAR-LL2-FILTER block sums the data which coming from OUT-TO-HAAR-16bytes[255..O] which represent the values of the pixels p(m , n) of the 4x4 window centred onto the scrolling 16x16 matrix, implemented by the SHIFTREGISTERS.
In this way, after a latency of 2 clock cycles, HAAR_LL2_FILTER produces a coefficient (expressed by 12 bits) per block cycle and it sends it to the 1LEV_MLPN_CLASSIFIER block through H_LL2[11..O].
As it has been described previously, the calculation which the MLPN classifier must perform are (for both the DC and HC classifiers):
followed by:
Figure imgf000016_0001
Due to the hardware high costs, necessary for implementing the activation function f(x) - see eq. (E16) -, it is preferable implementing in 1 LEV_MLPN_CLASSIFIER the equations:
, ^ , (E23)
D_xk = D_biask +∑D_wmrkD_nm m=0 H_xk = H _b
Figure imgf000017_0001
for /c=0..9.
The equations (E23) and (E231) represent the arguments of the activation function of (E21) and (E211). By doing this, such arguments are calculated in hardware and are sent to the host, which in software calculates f(p_xk) , f{H_xk) , (E22), and (E221). However, (E23) and (E231) represent 3,000 multiplications and 3,000 sums which are calculated in hardware, against 20 multiplications, 20 sums, 22 activation functions and the comparison to the threshold (E 17), calculated in the software by the host.
In order to perform this operation, 1LEV_MLPN_CLASSIFIER has been equipped with two sets of 10 Multiplier-and-ACcurnulators (MACs), that is, D_MAC,< and H_MACk (/c=0..9).
As soon as a coefficient D_LL2(i, j) [ H_LL2(i, J) ] is produced by
DAUB_LL2_FILTER [HAAR_LL2_FILTER ], the multipliers DJvlAC/c [ H_MACΛ ] multiply it in parallel times D_wm<k [H _wm k] (m=25/+/, k=0..9) and continue in this way for 150 clock strokes, a clock stroke (cc) for each one of the 150 coefficients of D_LL2 [ HJLL2 ].
The weights D__wm ' Λ and H_wm khave been previously stored in 20 LUTs during the setup (a LUT for each multiplier, which stores the 150 weights). The accumulator of each D_MACfc [ H_MACΛ] is initialized with D_biask [ H_hiask ] and it accumulates the products as soon as they are calculated by the multipliers.
Due to the latency, the operation of the 1LEV_MLPN_CLASSIFIER block ends up after 5 ccs from the last coefficients, D_LL2(5, 24) and H_LL2(5, 24), provided by DAUB_LL2_FILTER and by HAAR_LL2_FILTER. At this point, the data stored in the 20 accumulators of D_MACfe and of H_MACfc (/c=0..9) have now 63 bits and 45 bits, respectively, due to the increase in the dynamics. They are sent to the OUTPUTJNTERFACE block by means of DC_OUT_63bits_X_10neurons[629..0] and HC_OUT_45bits_X_10neurons[449..0 ].
These data are extended in sign and formatted in 64-bit words by the
OUTPUTJNTERFACE block. Furthermore, OUTPUTJNTERFACE serializes them by using a FIFO and it generates the irq signal which indicates that the results are ready to be collected. To end up, the host requires these results (read signal) and receives them on the dataread[63..O] (1 word/cc).
The following Table 1 sums-up the resource engagement required by the architecture. The Table shows also a comparison between the resources available on the Stratix™ EP1S60F1020C6 FPGA. In the partial use of the resources it is to be considered that also the previously described RD&TB block must be implemented on the same FPGA.
TABLE 1
USED RESOURCES
Used Available Use resources resources
Total logical elements 31,577 57,120 55%
Total pins 149 1 ,020 15%
Total memory bits 71 ,738 5,215,104 1%
DSP blocks 10 18 55%
Description of the experimental results and of the performances
In order to design and verify the processing core of the system according to the present invention, a sequence of rail images for about 9 Km has been acquired.
First of all, a known Error Back Propagation procedure with "adaptive learning rate" has been used to determine the biases and the classifier's weights. The image set adopted for training the neural network of the classifier includes 391 positive examples of hexagonal bolts with different orientation, and 703 negative examples represented by 24x100-pixel windows exctracted from the acquired video sequence.
The remaining video sequence has been used to perform the following tests. In defining the pre-processing strategy it has been observed that, although the DC classifier, based upon the DWT of Daubechies, has reached a very high detection rate (see part VII. C), nevertheless it has produced a certain number of false positives (FP) during the search in exhaustive phase.
In order to reduce these errors, a cross validation phase has been introduced. Then, the Haar DWT has been tested, given its scarse computational weight. The
HC classifier is a neural classifier which operates with the sub-band LL2 produced by the Haar DWT, and it has been designed and trained. HC reaches the same detection rate of DC, even if it detects several more false positives.
However, the false positives detected by HC derive from different features (windows) with respect to the false positives produced by DC. This phenomenon is underlined in figure 13C, wherein the result of the AND function applied between the detections (both real and false positives) obtained by DC (figure 13A) and by HC (figure 13B) is shown, when they have processed, in exhaustive phase (that is, without jump between the bolt pairs) 4,500 lines of the acquired video sequence.
As it results clear, only two false positives have been detected, with respect to analyzed 4,500 lines (analyzed 90,000 features), by means of the cross validation obtained through the AND function of DC and HC. The numerical results are shown in the following Table 2.
It can be noted that the FP/TP relationship is linked to the exhaustive phase, but it is reduced drastically during the jump-like search, which involves more than 98% of the processed lines (see section VILE).
TABLE 2
FALSE POSITIVES (EXHAUSTIVE SEARCH)
Real False Positives FP/TP FP/Analyzed positives (FP) lines
(TP)
Haar DWT 22 (100%) 90 409% 200.0%00
Daubechies DWT 22 (100%) 26 118% 57.8 %00
AND 22 (100%) 2 9% 4.4 %00
(Daubechies,
Haar)
In order to measure the accuracy of the system in detecting the presence/absence of the bolts, a prototype of the wholly software system has been carried out, which adopts precision of "floating point" type, in "trace" mode which allows to an observer to control the correctness of the automatic detections.
This experiment has been carried out for a sequence of 3,350 bolts. The system has detected 99.9% of visible bolts, 0.1% of hidden bolts and 95% of absent ones, as shown in the following Table 3.
TABLE 3
ACCURACY
Floating Point Fixed Point
Number of examined bolts 3,350 3,350 Number of visible bolts 2,649 2,649
Detected 2,646 (99.9%) 2,638 (99.6%)
Number of hidden bolts 721 721
Detected 1 (0.1%) 1 (0.1%)
Number of absent bolts 21 21
Detected 20 (95%) 20 (95%)
The report (file log) obtained by the above-mentioned test has been used as comparison term for the reports of similar experiments, which aim at defining the number of bits of the words to be used in the hardware design. The software prototype has been modified by floating point in fixed point. The different versions of the software procedures have been compiled with different precisions (that is number of used bits) both for the coefficients of the Daubechies filter and for the weights both of DC and HC. The setting with 23 bits for the filter coefficients and with 25 bits for the weights of both classifiers has produced a lower accuracy, in detecting the visible bolts, of only 0.3% than the one obtained using the precision in floating point. This compromise has been considered acceptable and the hardware has been developed by using these specifications.
Subsequent experimental tests have allowed to test the whole system (hardware and software) with the whole video sequence, to measure the reached calculation performances. The results of this test are shown in the following Table 4.
TABLE 4
OBTAINED PERFORMANCES
Processed lines 3,032,432 [lines] 9.097 [km]
Total time 215.34 [sec]
Speed 14,082 [lines/sec] 152.1 [km/h]
Jumped lines 2,980,012 [lines] 98.2% computational time of the jump-like search 159.93 [sec] 74.3% computational speed of the jump-like 18,633 [lines/sec] 201.2 [km/h] search lines processed in exhaustive way 52,420 [lines] 1.8% computational time of the exhaustive 55.41 [sec] 25.7% search computational speed of the exhaustive 946 [lines/sec] 10.2 [km/h] search
Pairs of examined bolts 15,027
These data derive from a software architecture developed in Visual C++, version 6.00 performed on a 3.2-GHz Pentium IV™ with 1 GB of the RAM which cooperates with the hardware architecture described in part Vl, clocked at 66 MHz and 100 MHz, which performs the analysis of a 24x100 window in 8.09μs (see figure 14).
The present invention has been sofar described according to a preferred embodiment thereof, shown by only way of example and not with limitative purpose. It is to be meant that other embodiments may be provided, all to be considered belonging to the protection scope of the same, as defined by the enclosed claims.

Claims

1. Automatic method for infrastructure visual inspection, for detecting characterizing features of the infrastructure itself, comprising the following steps of: acquiring images in sequence of subsequent portions of said infrastructure; - extracting from said acquired images geometrical coordinates of said infrastructure; - extracting from said acquired images, based upon said geometrical coordinates, sub-images corresponding to provided positions of said characterizing members; and - detecting in said sub-images the presence/absence of said characterizing members.
2. Method according to claim 1 , wherein said step of detecting in said sub- images the presence/absence of said characterizing members comprises: a phase of pre-processing said sub-images by means of at least a transform function; and a phase of classifying said sub-images pre-processed by means of at least a classifier, by determining the presence/absence of said characterizing members.
3. Method according to claim 1 or 2, wherein said step of extracting the geometrical coordinates from the acquired infrastructure images comprises a phase of data reduction in order to reduce the data to be handled, and a supervisioned classification phase, based upon neural network.
4. Method according to one of the preceding claims, wherein said step of extracting sub-images containing said characterizing members, comprises a phase of searching said characterizing members inside each one of said acquired images.
5. Method according to claim 4, wherein said searching phase comprises a searching mode of "exhaustive" type.
6. Method according to claim 4 or 5, wherein said searching phase comprises a second searching mode of "jump"-like type.
7. Method according to claims 5 and 6, wherein said searching phase is so as to alternate said first and second searching mode, depending upon the obtained results.
8. Method according to anyone of the claims 2 to 7, wherein said step of pre- processing said sub-images by means of at least a transform function provides the simultaneous application of two transforms of "bidimensional wavelet" type to the input data themselves.
9. Method according to claim 8, wherein a first one of said two transforms of "bidimensional wavelet" type is a Daubechies transform.
10. Method according to claim 9, wherein a second one of said two transforms of "bidimensional wavelet" type is a Haar transform.
11. Method according to claim 9, wherein said step of classifying said sub- images pre-processed by means of at least a classifier provides a first phase of classifying the results of said pre-processing by means of the Daubechies transform.
12. Method according to claim 10 or 11 , wherein said step of classifying said sub-images pre-processed by means of at least a classifier provides a second step of classifying the results of said pre-processing by means of Haar transform.
13. Method according to claims 11 and 12, wherein the results of said first and second classification phases are combined by means of a logical function.
14. Method according to claim 13, wherein said logical function is an AND function.
15. Method according to anyone of the claims 11 a 14, wherein said at least one classifier is a classifier of neural type.
16. Method according to anyone of the preceding claims, operating in real time.
17. Automatic automatized system of visual inspection of an infrastructure for detecting the characterizing members of the infrastructure itself, comprising an image acquisition unit (2) and a processing unit (3), wherein said processing unit (3) comprises: a first sub-unit (4) for extracting geometrical coordinates of said infrastructure from said acquired images; a second prediction sub-unit (5) for extracting from said acquired images, based upon said geometrical coordinates, sub-images corresponding to provided positions of said characterizing members; and - a third detection sub-unit (6) apt to determine the presence/absence of said characterizing members in said sub-images.
18. System according to claim 17, wherein said third detection sub-unit (6) comprises a module (7) for pre-processing said sub-images by at least a transform function and a module (7) for classifying said sub-images pre-processed by means of at least a classifier.
19. System according to claim 17 or 18, wherein said prediction sub-unit (5) comprises means for reducing the data and means for classifying said data, based upon neural network.
20. System according to one of the claims 17 to 19, wherein said prediction sub-unit (5) comprises means for searching said characterizing members inside each one of said acquired images.
21. System according to claim 20, wherein said means for searching said characterizing members inside each one of said acquired images is apt to perform a searching mode of "exhaustive" type.
22. System according to claim 20 o 21 , wherein said means for searching said characterizing members inside each one of said acquired images is apt to perform a searching mode of "jump"-like type.
23. System according to claims 21 and 22, wherein said means for searching said characterizing members inside each one of said acquired images is apt to alternate said first and second searching mode, depending upon the obtained results.
24. System according to any one of the claims 18 to 23, wherein said preprocessing module (7) comprises means (7', 7") in order to apply simultaneously two transforms of "bidimensional wavelet" type to the same input data.
25. System according to claim 24, wherein a first one of said two transforms of "bidimensional wavelet" type is a Daubechies transform.
26. System according to claim 24 or 25, wherein a second one of said two transforms of "bidimensional wavelet" type is a Haar transform.
27. System according to claims 24 and 25, wherein said classification module (8) comprises first means (8') in order to perform a first classification of the results of said pre-processing by means of the Daubechies transform.
28. System according to claim 24 e 26, wherein said classification module (8) comprises second means (8") in order to perform a second classification of the results of said pre-processing by means of a Haar transform.
29. System according to claims 27 and 28, wherein said classification module (8) comprises means for combining the results of said first and second classification by means of a logical function.
30. System according to claim 29, wherein said logical function is an AND function.
31. System according to any one of the claims 27 and 28, wherein said first and second classification means (8', 8") are classifiers of neural type.
32. System according to any one of the preceding claims, operating in real time.
33. System according to any one of the claims 17 to 31 , operating in real time.
PCT/IB2006/052438 2005-07-18 2006-07-17 Automatic method and system for visual inspection of railway infrastructure WO2007010473A2 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US11/988,449 US20090196486A1 (en) 2005-07-18 2006-07-17 Automatic Method and System for Visual Inspection of Railway Infrastructure
EP06780106A EP1904972A2 (en) 2005-07-18 2006-07-17 Automatic method and system for infrastructure visual inspection
JP2008522143A JP2009501855A (en) 2005-07-18 2006-07-17 Method and system for visual visual inspection of infrastructure
US13/564,551 US20130170709A1 (en) 2005-07-18 2012-08-01 Automatic method and system for visual inspection of railway infrastructure

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
ITRM2005A000381 2005-07-18
IT000381A ITRM20050381A1 (en) 2005-07-18 2005-07-18 METHOD AND AUTOMATIC VISUAL INSPECTION SYSTEM OF AN INFRASTRUCTURE.

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US13/564,551 Continuation US20130170709A1 (en) 2005-07-18 2012-08-01 Automatic method and system for visual inspection of railway infrastructure

Publications (2)

Publication Number Publication Date
WO2007010473A2 true WO2007010473A2 (en) 2007-01-25
WO2007010473A3 WO2007010473A3 (en) 2007-05-10

Family

ID=37560713

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2006/052438 WO2007010473A2 (en) 2005-07-18 2006-07-17 Automatic method and system for visual inspection of railway infrastructure

Country Status (5)

Country Link
US (2) US20090196486A1 (en)
EP (1) EP1904972A2 (en)
JP (1) JP2009501855A (en)
IT (1) ITRM20050381A1 (en)
WO (1) WO2007010473A2 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103902987A (en) * 2014-04-17 2014-07-02 福州大学 Station caption identifying method based on convolutional network
WO2022023594A1 (en) 2020-07-31 2022-02-03 Arborea Intellbird Sl System and method for infrastructure inspection

Families Citing this family (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ES2342621T3 (en) * 2006-12-13 2010-07-09 Ethicon Endo-Surgery, Inc. STORAGE OF BIOPSY SAMPLES.
US8345948B2 (en) * 2009-09-11 2013-01-01 Harsco Corporation Automated turnout inspection
US9280907B2 (en) 2009-10-01 2016-03-08 Kryterion, Inc. Proctored performance analysis
US9141513B2 (en) 2009-10-01 2015-09-22 Kryterion, Inc. Maintaining a secure computing device in a test taking environment
US10672286B2 (en) 2010-03-14 2020-06-02 Kryterion, Inc. Cloud based test environment
JP2011214933A (en) * 2010-03-31 2011-10-27 Kawasaki Heavy Ind Ltd Distance-image acquisition system for track
US8713130B2 (en) 2010-08-04 2014-04-29 Kryterion, Inc. Peered proctoring
US9137163B2 (en) 2010-08-04 2015-09-15 Kryterion, Inc. Optimized data stream upload
US8625878B2 (en) * 2011-04-15 2014-01-07 International Business Machines Corporation Method and system of rail component detection using vision technology
US9846025B2 (en) * 2012-12-21 2017-12-19 Wabtec Holding Corp. Track data determination system and method
US20140272882A1 (en) * 2013-03-13 2014-09-18 Kryterion, Inc. Detecting aberrant behavior in an exam-taking environment
US10349491B2 (en) 2015-01-19 2019-07-09 Tetra Tech, Inc. Light emission power control apparatus and method
US9849895B2 (en) 2015-01-19 2017-12-26 Tetra Tech, Inc. Sensor synchronization apparatus and method
CA2892952C (en) 2015-01-19 2019-10-15 Tetra Tech, Inc. Protective shroud
KR102339517B1 (en) * 2015-01-27 2021-12-16 현대모비스 주식회사 Ultrasonic Wave Sensing Apparatus and Method Thereof for Vehicle
CA2892885C (en) 2015-02-20 2020-07-28 Tetra Tech, Inc. 3d track assessment system and method
US10311551B2 (en) 2016-12-13 2019-06-04 Westinghouse Air Brake Technologies Corporation Machine vision based track-occupancy and movement validation
US11078631B2 (en) 2017-11-30 2021-08-03 Sperry Rail Holdings, Inc. System and method for inspecting a rail using machine learning
US10730538B2 (en) 2018-06-01 2020-08-04 Tetra Tech, Inc. Apparatus and method for calculating plate cut and rail seat abrasion based on measurements only of rail head elevation and crosstie surface elevation
US11377130B2 (en) 2018-06-01 2022-07-05 Tetra Tech, Inc. Autonomous track assessment system
US10625760B2 (en) 2018-06-01 2020-04-21 Tetra Tech, Inc. Apparatus and method for calculating wooden crosstie plate cut measurements and rail seat abrasion measurements based on rail head height
US10807623B2 (en) 2018-06-01 2020-10-20 Tetra Tech, Inc. Apparatus and method for gathering data from sensors oriented at an oblique angle relative to a railway track
WO2020172386A1 (en) * 2019-02-20 2020-08-27 International Electronic Machines Corp. Machine vision based inspection
CN110097536B (en) * 2019-04-10 2023-04-18 东南大学 Hexagonal bolt looseness detection method based on deep learning and Hough transform
AU2020273465A1 (en) 2019-05-16 2022-01-06 Tetra Tech, Inc. System and method for generating and interpreting point clouds of a rail corridor along a survey path

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6683974B1 (en) * 2000-01-12 2004-01-27 Sharp Kabushiki Kaisha Image defect detection apparatus and image defect detection method
US20040183900A1 (en) * 2003-03-20 2004-09-23 Everest Vit Method and system for automatically detecting defects in remote video inspection applications

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ITRM20010045A1 (en) * 2001-01-29 2002-07-29 Consiglio Nazionale Ricerche SYSTEM AND METHOD FOR DETECTING THE RELATIVE POSITION OF AN OBJECT COMPARED TO A REFERENCE POINT.
US7305371B2 (en) * 2001-07-06 2007-12-04 Newvectors Llc Swarming agents for distributed pattern detection and classification
JP3785456B2 (en) * 2002-07-25 2006-06-14 独立行政法人産業技術総合研究所 Safety monitoring device at station platform
GB0222211D0 (en) * 2002-09-25 2002-10-30 Fortkey Ltd Imaging and measurement system
TWI274843B (en) * 2002-12-25 2007-03-01 Hon Hai Prec Ind Co Ltd System and method for obtaining measurement data on objects via processing images thereof

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6683974B1 (en) * 2000-01-12 2004-01-27 Sharp Kabushiki Kaisha Image defect detection apparatus and image defect detection method
US20040183900A1 (en) * 2003-03-20 2004-09-23 Everest Vit Method and system for automatically detecting defects in remote video inspection applications

Non-Patent Citations (8)

* Cited by examiner, † Cited by third party
Title
ALIPPI C ET AL: "An embedded system methodology for real-time analysis of railways track profile" IMTC/2002. PROCEEDINGS OF THE 19TH IEEE INSTRUMENTATION AND MEASUREMENT TECHNOLOGY CONFERENCE (IEEE CAT. NO.00CH37276) IEEE PISCATAWAY, NJ, USA, vol. 1, 2002, pages 747-751 vol.1, XP002414285 ISBN: 0-7803-7218-2 *
ALIPPI C ET AL: "Composite real-time image processing for railways track profile measurement" IMTC/99. PROCEEDINGS OF THE 16TH IEEE INSTRUMENTATION AND MEASUREMENT TECHNOLOGY CONFERENCE (CAT. NO.99CH36309) IEEE PISCATAWAY, NJ, USA, vol. 2, 1999, pages 1283-1288 vol., XP002414286 ISBN: 0-7803-5276-9 *
BARKE D ET AL: "Structural health monitoring in the railway industry: A review" STRUCTURAL HEALTH MONITORING, [Online] vol. 4, no. 1, 1 March 2005 (2005-03-01), pages 81-93, XP002414288 Sage Publications, Thousand Oaks, CA, USA Retrieved from the Internet: URL:http://shm.sagepub.com/cgi/repring/4/1/81.pdf> [retrieved on 2007-01-10] *
CESARE ALIPPI ET AL: "Composite Real-Time Image Processing for Railways Track Profile Measurement" IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, IEEE SERVICE CENTER, PISCATAWAY, NJ, US, vol. 49, no. 3, June 2000 (2000-06), XP011024993 ISSN: 0018-9456 cited in the application *
CYBERNETIX LTD: "Cybernétix' Railway Activity" PRODUCT WHITE PAPER, [Online] September 2004 (2004-09), pages 1-4, XP002414287 Marseille, France Retrieved from the Internet: URL:http://www.cybernetix.fr/IMG/pdf/fiche_Railway_uk_A.pdf> [retrieved on 2007-01-10] *
MAZZEO P L ET AL: "Visual recognition of fastening bolts for railroad maintenance" PATTERN RECOGNITION LETTERS, NORTH-HOLLAND PUBL. AMSTERDAM, NL, vol. 25, no. 6, 19 April 2004 (2004-04-19), pages 669-677, XP004498191 ISSN: 0167-8655 *
RUBAAI A: "A NEURAL-NET-BASED DEVICE FOR MONITORING AMTRAK RAILROAD TRACK SYSTEM" IEEE TRANSACTIONS ON INDUSTRY APPLICATIONS, IEEE SERVICE CENTER, PISCATAWAY, NJ, US, vol. 39, no. 2, March 2003 (2003-03), pages 374-381, XP001158273 ISSN: 0093-9994 cited in the application *
SATO K ET AL: "Obstruction detector using ultrasonic sensors for upgrading the safety of a level crossing" DEVELOPMENTS IN MASS TRANSIT SYSTEMS, 1998. INTERNATIONAL CONFERENCE ON (CONF. PUBL. NO. 453) LONDON, UK 20-23 APRIL 1998, LONDON, UK,IEE, UK, 20 April 1998 (1998-04-20), pages 190-195, XP006503973 ISBN: 0-85296-703-9 cited in the application *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103902987A (en) * 2014-04-17 2014-07-02 福州大学 Station caption identifying method based on convolutional network
CN103902987B (en) * 2014-04-17 2017-10-20 福州大学 A kind of TV station symbol recognition method based on convolutional network
WO2022023594A1 (en) 2020-07-31 2022-02-03 Arborea Intellbird Sl System and method for infrastructure inspection

Also Published As

Publication number Publication date
JP2009501855A (en) 2009-01-22
US20130170709A1 (en) 2013-07-04
EP1904972A2 (en) 2008-04-02
WO2007010473A3 (en) 2007-05-10
ITRM20050381A1 (en) 2007-01-19
US20090196486A1 (en) 2009-08-06

Similar Documents

Publication Publication Date Title
EP1904972A2 (en) Automatic method and system for infrastructure visual inspection
Marino et al. A real-time visual inspection system for railway maintenance: Automatic hexagonal-headed bolts detection
Zhang et al. Real-time detection of cracks on concrete bridge decks using deep learning in the frequency domain
Nejad et al. An optimum feature extraction method based on wavelet–radon transform and dynamic neural network for pavement distress classification
Nejad et al. An expert system based on wavelet transform and radon neural network for pavement distress classification
Zhang et al. A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application
Mazzeo et al. Visual recognition of fastening bolts for railroad maintenance
Zhang et al. Multi-focus image fusion algorithm based on focused region extraction
Yousaf et al. Comparative analysis of automatic vehicle classification techniques: a survey
CN102254314A (en) Visible-light/infrared image fusion method based on compressed sensing
CN104463104A (en) Fast detecting method and device for static vehicle target
Tabatabaei et al. Automatic detection of the cracks on the concrete railway sleepers
Tang et al. Dual-attention-based feature aggregation network for infrared and visible image fusion
CN108921147A (en) A kind of black smoke vehicle recognition methods based on dynamic texture and transform domain space-time characteristic
De Ruvo et al. A FPGA-based architecture for automatic hexagonal bolts detection in railway maintenance
Nguyen et al. Analysis of the influence of de-hazing methods on vehicle detection in aerial images
De Ruvo et al. A visual inspection system for rail detection & tracking in real time railway maintenance
Paygude et al. Unified dynamic texture segmentation system based on local and global spatiotemporal techniques
Gang et al. Infrared and visible dynamic image sequence fusion based on region target detection
Le-Tien et al. Combined Zernike moment and multiscale Analysis for tamper detection in digital images
Liu et al. A dynamic weights-based wavelet attention neural network for defect detection
Harb et al. Histogram of oriented gradients and texture features for bone texture characterization
Williams et al. A family of algorithms for the automatic detection, isolation, and fusion of object responses in sonar data
Visshwak et al. On-the-fly traffic sign image labeling
Mazzeo et al. Visual recognition of fastening bolt in railway maintenance context by using wavelet transform

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application
WWE Wipo information: entry into national phase

Ref document number: 2008522143

Country of ref document: JP

Ref document number: 2006780106

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

WWW Wipo information: withdrawn in national office

Ref document number: DE

WWP Wipo information: published in national office

Ref document number: 2006780106

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 11988449

Country of ref document: US