Classification of Point Clouds with Neural Networks and Continuum-Type Memories

. This paper deals with the issue of evaluating and analyzing geometric point sets in three-dimensional space. Point sets or point clouds are often the product of 3D scanners and depth sensors, which are used in the ﬁeld of autonomous movement for robots and vehicles. Therefore, for the classiﬁcation of point sets within an active motion, not fully generated point clouds can be used, but knowledge can be extracted from the raw impulses of the respective time points. Attractors consisting of a continuum of stationary states and hysteretic memories can be used to couple multiple inputs over time given non-independent output quantities of a classiﬁer and applied to suitable neural networks. In this paper, we show a way to assign input point clouds to sets of classes using hysteretic memories, which are transferable to neural networks.


Introduction
Depth sensors have become ubiquitous in many application areas, e.g.robotics, driver assistance systems, geo modeling, and 3D scanning using smartphones.Depth sensing is achieved by means of waves or rays that are sent out by a transmitter, reflected at surfaces and detected again by an receiver.The time difference between emitting and receiving gives information about the distance covered.Various physical principles are used, such as electromagnetic waves (radar), acoustic waves (sonar) or laser beams (LiDAR).In our research, we aim at developing an autonomous robotic platform that serves as a carrier for various LiDAR and sonar sensors for 3D mapping and environmental data acquisition in inland waters [2].3D point clouds obtained from ultrasonic and LiDAR mapping need to get semantically segmented and classified through machine learning analysis to extract knowledge for object detection and autonomous movement.In order to train such classifiers, however, large amounts of training data are required that provide labeled examples of correct classifications.Therefore we developed an approach where virtual worlds with virtual depth sensors are used to generate labeled point clouds (fig. 1 [16]).To prepare an application in certain use cases, AI algorithms (e.g.neural networks) are trained with synthetic data obtained in virtual environments under various conditions and signal qualities [16].
Fig. 1.Exemplary synthetic labeled point cloud of virtual LiDAR sensor with ground truth data for part segmentation of 3D chair meshes.The axes represent independent units of measurement of the virtual environment (Blender 3D software suite).
However, a distinction must be made here between a completely registered point cloud (as depicted in fig. 1) and raw temporal impulses from the sensors.For active use in robotic movement and automization, the raw temporal impulses must be evaluated, while the completely registered point cloud is created in postprocessing, but initially offers no added value for the motion behavior.Therefore, ways have to be found to extract knowledge for object recognition and classification from the raw temporal impulses, as these are the data the robot gets at a certain point of time.
This approach differs from a variety of considerations on how machine learning can be used to classify point clouds (see sec 2.2).In order to use temporal impulses for classification in a meaningful way, it must be possible to link several impulses of different points of time with each other.Accordingly, it must be possible to link historical output variables with each other via a memory.The classifier is thus not solely dependent on the input variables (sensor pulses), but also on the previous state of the output variable.The system can thus -depending on the previous history -assume one of several possible states for the same input variable.These considerations lead us to the assumption that hysteresis memories and neural network applications can be coupled with each other.

Background
This section details the fundamental concepts of light and sound propagation and reflection under ideal and perturbed conditions.

Depth Sensing
LiDAR & Time-of-Flight (ToF) LiDAR is a common method for optical distance measurement.Generally, passive and active sensor systems can be distinguished.In this article we focus on active distance measurement, where radiation is introduced into the environment by the measuring device.An example are cameras that can determine distances by means of ToF or the active triangulation [11].
Sonar / Ultrasonic Sonar is a distance measurement method based on sound waves.Like LiDAR, measurements devices can be divided into active and passive 3 Sonar.Again, only the more common active type is considered in this paper.Here, the transmitter emits a signal in the form of a sound wave.The sound wave is reflected at the target object and registered at the receiver.The time difference between signal transmission and reception provides information about the distance.The type can be one of the options rotating, static, or side-scan.Depending on the sensor, different configuration options are available for the sensor's field of view.The first two sensors have a field of view in horizontal and vertical direction, while the side-scan sensor only has a downward opening angle.Active sonars are used, for example, to locate schools of fish in waters or to map underwater stuctures (see fig. 2).The task is now to be able to correctly assign the points to object classes with an arbitrary classification function f .However, since there is no complete point cloud, but only individual pulses, the object classes must be detected via the connection of several time steps.This approach has different requirements than those found, for example, in [13].In the following section, we therefore discuss previous approaches and work out the requirements for the model.

Classification and Semantic Segmentation
Following the assumptions of [14], we can summarize the main properties of the input point clouds or a subset of them: -Unordered: points in a point cloud are typically not assumed to have any particular structure -Density variability: non-constant density within the point clouds due to perspective effects, movements or measurement errors -Invariance under transformation: as a set, such data must be invariant to mutations of its members One option for automated processing of point clouds with machine learning, e. g. for segmentation, is unsupervised learning.For example, a cluster analysis can be used to segment a point cloud into certain parts.Such a procedure is described in [4] and [3].The classification of data points into groups (also: clusters) is done by grouping elements that are as similar as possible to each other.More detailed information on this can be found in [1].
Supervised learning is usually used for classification tasks and, in contrast to unsupervised learning, requires semantically labeled examples for training.For 2D image data, the learning of classifies from examples is described e.g. or [15].Learning of classifiers for the case of three-dimensional data has been investigated e.g. in [13], [14] and [24].Regardless of the number of dimensions, methods of supervised learning are often used to automatically recognize patterns and relationships.Such an approach, however, requires a large amount of training data with the correct classification for each pixel or point in a scene (pixel-wise or point-wise segmentation).Alternatively, smallest circumscribing rectangles or bounding boxes may be used depending on the needed amount of precision.
So various applications have shown that the use of neural networks in the analysis of LiDAR and sonar data yields promising results for classification and semantic segmentation.However, for most of the methods a pre-processing of the pulse signals derived from LiDAR sensors is necessary, which results in high computational overhead and considerable latency.Spiking Neural Networks (SNN), referred as third generation neural networks, showed promising results in modeling time dependent data as depth sensing signals (see 2.1) are.SNN are capable to compute the raw temporal impulses of LiDAR and sonar and are so meaningful in the real-time automization of robotic movements.Using the synthetic raw pulse data of simulated depth sensors derived in the virtual environments (like in fig.2), we want to address the object recognition problem with SNN on raw temporal pulses.
However, both conventional Deep Learning structures and SNN suffer from a nontransparent representation from an input set to an output class.To extend these capabilities, hysteresis memories can be used, which can address the described classification problem through convergence behavior.
3 Continuum stationary sets in an acoustic-ODE problem

Stability investigations by abstract Volterra integral equations
In this section we consider a boundary control problem for the interactions of acoustic waves with some control processes on the boundary described by an ODE.Note that on the level of time-series this coupling problem can arise as temporal coding in SNN [6].Our aim is to provide frequency-domain conditions for the convergence of trajectories of the process to the stationary set (attractor) which consists of a continuum of functions.In order to show this we introduce some function spaces needed in the sequel and consider a general Volterra integral equation.The problem of convergence of solutions of the PDE/ODE problem will be reduced to the investigation of such an integral equation.
The interaction of acoustic and laser signals with a structure (control or heating part) can be described as hybrid system consisting of different types of PDE's and ODE's connected by special boundary conditions.In the case of a microwave heating problem the incoming signal is given by Maxwell's equations, the heating process is defined by parabolic equations.Under certain assumptions in this situation a multivalued dynamical system having a global attractor is generated [19].
( ḟ denotes the distribution derivative.)Assume that the map t ∈ R + → L (U, Z) is twice piecewise-differentiable and satisfies the following condition: There exists a ρ 0 > 0 and a constant γ > 0 such that and Consider the Volterra integral equation where K(t) ∈ L (U, Z)(U, Z Hilbert spaces ) is twice piecewise-differentiable satisfies (3.1) and (3.2), and has therefore a state-space realization.Suppose that φ : Z ×R + → U is a continuous function.
Instead of one fixed nonlinearity φ we consider a family N of continuous maps, such that for any φ ∈ N and any h ∈ D(A) with D(A) from the nonlinear integral equation (3.3) has a unique solution z(•, h, φ ) and this solution is continuous.
In order to describe the absolute stability or instability behavior of (3.3) in the following theorem we need an additional assumption on the class N .Let us assume that there is a linear bounded operator R : Z → U such that the "nonlinearity" ϕ(z) = Rz belongs to N .Theorem 3.1.( [17]) Suppose that χ(•) is the Laplace transform of K and the operator function (I − χ(λ )R) −1 has poles in the right half-plane and the frequency-domain condition from [18] is satisfied.Then there exists a bounded linear self-adjoint operator −ρ (0, +∞; Z) with the following properties: a) There exists a constant β > 0 such that for any h ∈ C and any φ ∈ N 3) which does not satisfy (3.4) has the property In the next subsection we show an application of this theorem.

Hyperbolic PDE's with boundary control as realizations of Volterra equations
Let us investigate the question how to suppress vibrations in a fluid conveying tube via control on the boundary.We consider for this a system of equations which is described in [12,21].The motion of an incompressible fluid is given for t > 0 in the acoustic approximation by the hyperbolic PDE's where a 1 and a 2 are positive parameters, v denotes the relative velocity of the fluid and w denotes the pressure.The boundary conditions are given for t > 0 by where u(•) is a function ("boundary control") which describes the relative displacement of the piston of a servomotor.The equation of the turbine for t > 0 is Here q denotes the relative angular speed of the turbine, T a is a positive parameter.The regulator is described by the equation where ζ represents the displacement of the clutch of the regulator and T r , T k , c 0 and k are positive parameters.The friction term is given by a continuous function φ : R → R defined through a parameter κ > 0 by The equation of the servomotor is where T s is a positive parameter and η denotes the displacement of the slide value.The last condition is for t > 0 and with a positive parameter β η where . a) Note that χ(λ ) can be written with some c > 0 as The representation (3.13) shows that χ 1 (λ ) is analytic in some halfplane {Re λ > −ε} with ε > 0. From this it follows that χ 1 (λ ) has the Laplace original K 1 (t) which is absolute continuous, satisfies the inequalities with some ε 0 > 0 and such that K 1 and K1 belong to L 2 (0, ∞; R).Since the first part of (3.13) has the Laplace original k T 2 r e −ct/T 2 r the whole original of χ(λ ) can be represented as It is shown in [21] that any solution component z(t) := ζ (t) from (3.9) can be written as where again h is absolute continuous, satisfies an inequality of type (3.14) and h, ḣ belong to L 2 (0, ∞; R).The quadratic constraints can be described in Using the transfer function (3.13) and the constraints (3.16) we can verify the frequency-domain condition from [18].A direct computation shows (see [21]) that if is necessary and sufficient for the frequency-domain condition.The stability and instability domains of the denominator of χ(λ ) where investigated in [12] and characterized in the (T k , T Particularly one can derive frequency-domain conditions for the point-wise strong (in the sense of L p ) convergence of solutions of the above fluid control system to the stationary set (attractor) consisting of a continuum of functions ( [21]).
4 Convergence to the stationary set in systems with hysteresis

Evolutionary variational inequalities
It is widely recognized that the Dirac delta function δ plays an important role in neural network theory, e.g., for the description of integrate and fire neurons.We now show one way of introducting δ into the framework of nonlinear systems with hysteresis nonlinearity, which can be used for constructing continuum-type memories.Some applications of hysteretic memories are demonstrated in [8,23].The multivalued properties of hysteresis operators are described by variational inequalities or differential inclusions ( [22]) which are defined in some abstract setup.
Suppose that Y 0 is a real Hilbert space with (•, •) 0 and • 0 as scalar product resp.norm.Suppose also that A : D(A) ⊂ Y 0 is a closed (unbounded) densely defined linear operator.The Hilbert space Y 1 is defined as D(A) equipped with the scalar product If −∞ ≤ T 1 < T 2 ≤ +∞ are arbitrary numbers, we define the norm for Bochner measurable functions in L 2 (T 1 , T 2 ;Y j ) , j = 1, 0, −1 , through y 2, j := ( T 2 T 1 y(t) 2 j dt) 1/2 .For an arbitrary interval J in R denote by W (J ) the space of functions y(•) ∈ L 2 loc (J ;Y 1 ) for which ẏ (•) ∈ L 2 loc (J ;Y −1 ) equipped with the norm defined for any u(t) = φ (z, u 0 )(t), z(t) = Cy(t), y(0) = y 0 ∈ Y 0 , u 0 ∈ E (z(0)).Note that parabolic equations with Dirac's delta function at the right-hand side arise also as Fokker-Planck-Kolmogorov equation for the probability density of neurons in nonlinear noisy leaky integrate and fire models of neural networks [5].Convergence to the stationary set is connected with the construction of global attractors as working memory [7,9,20].General Hausdorff dimension and stability properties of such attractors for smooth and non-smooth dynamical systems and cocycles (non-autonomous dynamical systems) are considered in [10].

Conclusion
With this paper, we could present an approach to bring continuum-type memories and neural networks together.The convergence behavior of continuum-type memories in the representation of a trajectory from input data set to an output class can be used to improve the understanding and further analysis for neural networks.For distance measurement methods, which are mainly represented by temporally offset pulses that can be better recognized by specific neural networks (e.g., spiking neural networks), this is a subject for further research to be verified in experiments.

Fig. 2 .
Fig. 2. Temporal pulses of side scan sonar in virtual underwater environment.Red points are grouped to the underground, blue points to irregularities, e. g. certain underwater objects.Each segment compries one sonar pulse at one certain point of time.Again the axes represent independent units of measurement of the virtual environment (Blender).

2 ,
For a Hilbert space Y with scalar product (•, •) and norm | • | the space L 2 loc (R;Y ) consists of locally L 2 -functions on R with values in Y and with a topology defined by the family of seminorms | y | n := n −n | y(t) | 2 dt 1/n = 1, 2, . . . .Thus the space L 2 loc (R;Y ) is considered as a Fréchet space, i.e. as a complete metrizable linear topological space.For any interval J ⊂ R we regard L 2 loc (J ; Y ) as a subspace of L 2 loc (R;Y ) identifying L 2 loc (J ;Y ) with the set of functions in L 2 loc (R;Y ) which vanish outside of J .For any interval J ⊂ R, a Hilbert space Y and a parameter ρ ∈ R we introduce the weighted spaces L 2 ρ (J ;Y ) and W 1,2 ρ (J ;Y ) by .11) A direct computation shows that the transfer function of the linear part of (3.6) -(3.11) which connects the (formal) Laplace transforms of −φ and ζ is given by 2 r )-plane by domains Ω st and Ω unst , respectively.It follows now that under the conditions (3.16) -(3.18) for parameters from Ω st or Ω unst the solutions of the integral equation (3.15) (and the solutions of the PDE problem (3.6) -(3.11)) have the properties described by Theorem 3.1.
(y, η) 1 := ((β I − A)y, (β I − A)η) 0 , y, η ∈ D(A) , where β ∈ ρ(A) (ρ(A) is the resolvent set of A) is an arbitrary but fixed number the existence of which we assume.The Hilbert space Y −1 is by definition the completion of Y 0 with respect to the norm z −1 = (β I − A) −1 z 0 .Thus we have the dense and continuous embedding Y 1 ⊂ Y 0 ⊂ Y −1 which is called Hilbert space rigging structure.The duality pairing (•, •) −1,1 on Y 1 × Y −1 is the unique extension by continuity of the functionals (•, y) 0 with y ∈ Y 1 onto Y −1 .

Part 3
and 4 of the work are supported by the 2020-2021 program Leading Scientific Schools of the Russian Federation (project NSh-2624.2020.1)and Saint Petersburg State University (ID 75206671).