{"topic":"ai-mathematics","sort":"top","limit":25,"offset":0,"count":25,"papers":[{"id":"arxiv:1706.02515","kind":"preprint","title":"Self-Normalizing Neural Networks","authors":["Günter Klambauer","Thomas Unterthiner","Andreas Mayr","Sepp Hochreiter"],"author_count":4,"year":2017,"published":"2017-06-08","venue":"Advances in Neural Information Processing Systems 30 (NIPS 2017)","cited_by":2031,"doi":null,"arxiv_id":"1706.02515","nct_id":null,"source_url":"http://arxiv.org/abs/1706.02515","archive_url":"https://space-kitty.com/papers/arxiv:1706.02515","updated":"2017-09-07","pmid":null,"pmcid":null,"openalex_id":"W2624413595","pdf_url":"https://arxiv.org/pdf/1706.02515","abstract":"Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are \"scaled exponential linear units\" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Neural Networks and Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1038/s41586-023-06747-5","kind":"article","title":"Solving olympiad geometry without human demonstrations","authors":["Trieu H. Trinh","Yuhuai Wu","Quoc Viet Le","He He","Minh-Thang Luong"],"author_count":5,"year":2024,"published":"2024-01-17","venue":"Nature","cited_by":311,"doi":"10.1038/s41586-023-06747-5","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1038/s41586-023-06747-5","archive_url":"https://space-kitty.com/papers/doi:10.1038/s41586-023-06747-5","updated":null,"pmid":"38233616","pmcid":null,"openalex_id":"W4390940921","pdf_url":"https://www.nature.com/articles/s41586-023-06747-5.pdf","abstract":"Abstract Proving mathematical theorems at the olympiad level represents a notable milestone in human-level automated reasoning 1–4 , owing to their reputed difficulty among the world’s best talents in pre-university mathematics. Current machine-learning approaches, however, are not applicable to most mathematical domains owing to the high cost of translating human proofs into machine-verifiable format. The problem is even worse for geometry because of its unique translation challenges 1,5 , resulting in severe scarcity of training data. We propose AlphaGeometry, a theorem prover for Euclidean plane geometry that sidesteps the need for human demonstrations by synthesizing millions of theorems and proofs across different levels of complexity. AlphaGeometry is a neuro-symbolic system that uses a neural language model, trained from scratch on our large-scale synthetic data, to guide a symbolic deduction engine through infinite branching points in challenging problems. On a test set of 30 latest olympiad-level problems, AlphaGeometry solves 25, outperforming the previous best method that only solves ten problems and approaching the performance of an average International Mathematical Olympiad (IMO) gold medallist. Notably, AlphaGeometry produces human-readable proofs, solves all geometry problems in the IMO 2000 and 2015 under human expert evaluation and discovers a generalized version of a translated IMO theorem in 2004.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["AI-based Problem Solving and Planning"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Solving olympiad geometry without human demonstrations\" by Trieu H. Trinh, Yuhuai Wu, Quoc Viet Le et al., https://doi.org/10.1038/s41586-023-06747-5, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"openalex:W2963097630","kind":"article","title":"The expressive power of neural networks: a view from the width","authors":["Lu Zhou","Hongming Pu","Feicheng Wang","Zhiqiang Hu","Liwei Wang"],"author_count":5,"year":2017,"published":"2017-12-04","venue":"neural information processing systems","cited_by":516,"doi":null,"arxiv_id":null,"nct_id":null,"source_url":"https://papers.nips.cc/paper/2017/file/32cbf687880eb1674a07bf717761dd3a-Paper.pdf","archive_url":"https://space-kitty.com/papers/openalex:W2963097630","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W2963097630","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Neural Networks and Applications"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1109/tsmcb.2012.2226577","kind":"article","title":"Direct Adaptive Neural Control for a Class of Uncertain Nonaffine Nonlinear Systems Based on Disturbance Observer","authors":["Mou Chen","Shuzhi Sam Ge"],"author_count":2,"year":2012,"published":"2012-12-05","venue":"IEEE Transactions on Cybernetics","cited_by":396,"doi":"10.1109/tsmcb.2012.2226577","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tsmcb.2012.2226577","archive_url":"https://space-kitty.com/papers/doi:10.1109/tsmcb.2012.2226577","updated":null,"pmid":"26502431","pmcid":null,"openalex_id":"W1992171281","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Adaptive Control of Nonlinear Systems"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:1705.09886","kind":"preprint","title":"Convergence Analysis of Two-layer Neural Networks with ReLU Activation","authors":["Yuanzhi Li","Yuan Yang"],"author_count":2,"year":2017,"published":"2017-05-28","venue":null,"cited_by":304,"doi":null,"arxiv_id":"1705.09886","nct_id":null,"source_url":"http://arxiv.org/abs/1705.09886","archive_url":"https://space-kitty.com/papers/arxiv:1705.09886","updated":"2017-11-01","pmid":null,"pmcid":null,"openalex_id":"W2618398196","pdf_url":"https://arxiv.org/pdf/1705.09886","abstract":"In recent years, stochastic gradient descent (SGD) based techniques has become the standard tools for training neural networks. However, formal theoretical understanding of why SGD can train neural networks in practice is largely missing. In this paper, we make progress on understanding this mystery by providing a convergence analysis for SGD on a rich subset of two-layer feedforward networks with ReLU activations. This subset is characterized by a special structure called \"identity mapping\". We prove that, if input follows from Gaussian distribution, with standard $O(1/\\sqrt{d})$ initialization of the weights, SGD converges to the global minimum in polynomial number of steps. Unlike normal vanilla networks, the \"identity mapping\" makes our network asymmetric and thus the global minimum is unique. To complement our theory, we are also able to show experimentally that multi-layer networks with this mapping have better performance compared with normal vanilla networks. Our convergence theorem differs from traditional non-convex optimization techniques. We show that SGD converges to optimal in \"two phases\": In phase I, the gradient points to the wrong direction, however, a potential function $g$ gradually decreases. Then in phase II, SGD enters a nice one point convex region and converges. We also show that the identity mapping is necessary for convergence, as it moves the initial point to a better place for optimization. Experiment verifies our claims.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Stochastic Gradient Optimization Techniques"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1016/j.neunet.2020.01.018","kind":"article","title":"Theory of deep convolutional neural networks: Downsampling","authors":["Ding‐Xuan Zhou"],"author_count":1,"year":2020,"published":"2020-01-25","venue":"Neural Networks","cited_by":263,"doi":"10.1016/j.neunet.2020.01.018","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1016/j.neunet.2020.01.018","archive_url":"https://space-kitty.com/papers/doi:10.1016/j.neunet.2020.01.018","updated":null,"pmid":"32036229","pmcid":null,"openalex_id":"W3002335888","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Neural Networks and Applications"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2108.08481","kind":"preprint","title":"Neural Operator: Learning Maps Between Function Spaces","authors":["Nikola Kovachki","Zongyi Li","Burigede Liu","Kamyar Azizzadenesheli","Kaushik Bhattacharya","Andrew M. Stuart","Animashree Anandkumar"],"author_count":7,"year":2021,"published":"2021-08-19","venue":"The Journal of Machine Learning Research (2023), Volume 24, Issue 1, Article No 89, pp 4061-4157","cited_by":217,"doi":"10.5555/3648699.3648788","arxiv_id":"2108.08481","nct_id":null,"source_url":"http://arxiv.org/abs/2108.08481","archive_url":"https://space-kitty.com/papers/arxiv:2108.08481","updated":"2024-05-02","pmid":null,"pmcid":null,"openalex_id":"W3194436063","pdf_url":"https://arxiv.org/pdf/2108.08481","abstract":"The classical development of neural networks has primarily focused on learning mappings between finite dimensional Euclidean spaces or finite sets. We propose a generalization of neural networks to learn operators, termed neural operators, that map between infinite dimensional function spaces. We formulate the neural operator as a composition of linear integral operators and nonlinear activation functions. We prove a universal approximation theorem for our proposed neural operator, showing that it can approximate any given nonlinear continuous operator. The proposed neural operators are also discretization-invariant, i.e., they share the same model parameters among different discretization of the underlying function spaces. Furthermore, we introduce four classes of efficient parameterization, viz., graph neural operators, multi-pole graph neural operators, low-rank neural operators, and Fourier neural operators. An important application for neural operators is learning surrogate maps for the solution operators of partial differential equations (PDEs). We consider standard PDEs such as the Burgers, Darcy subsurface flow, and the Navier-Stokes equations, and show that the proposed neural operators have superior performance compared to existing machine learning based methodologies, while being several orders of magnitude faster than conventional PDE solvers.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Model Reduction and Neural Networks"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1016/j.neucom.2014.12.031","kind":"article","title":"Global stability analysis of fractional-order Hopfield neural networks with time delay","authors":["Wang Hu","Yongguang Yu","Guoguang Wen","Shuo Zhang","Junzhi Yu"],"author_count":5,"year":2014,"published":"2014-12-20","venue":"Neurocomputing","cited_by":258,"doi":"10.1016/j.neucom.2014.12.031","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1016/j.neucom.2014.12.031","archive_url":"https://space-kitty.com/papers/doi:10.1016/j.neucom.2014.12.031","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W2053751819","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Neural Networks Stability and Synchronization"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1109/tsmc.2018.2875947","kind":"article","title":"Adaptive Neural Backstepping Control Design for A Class of Nonsmooth Nonlinear Systems","authors":["Xudong Zhao","Xinyong Wang","Shuo Zhang","Guangdeng Zong"],"author_count":4,"year":2018,"published":"2018-12-13","venue":"IEEE Transactions on Systems Man and Cybernetics Systems","cited_by":208,"doi":"10.1109/tsmc.2018.2875947","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tsmc.2018.2875947","archive_url":"https://space-kitty.com/papers/doi:10.1109/tsmc.2018.2875947","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W2905413452","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Adaptive Control of Nonlinear Systems"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1109/tnn.2011.2109735","kind":"article","title":"Recurrent Neural Network for Non-Smooth Convex Optimization Problems With Application to the Identification of Genetic Regulatory Networks","authors":["Long Cheng","Zeng‐Guang Hou","Yingzi Lin","Min Han Tan","Wenjun Zhang","Fang‐Xiang Wu"],"author_count":6,"year":2011,"published":"2011-03-22","venue":"IEEE Transactions on Neural Networks","cited_by":213,"doi":"10.1109/tnn.2011.2109735","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tnn.2011.2109735","archive_url":"https://space-kitty.com/papers/doi:10.1109/tnn.2011.2109735","updated":null,"pmid":"21427022","pmcid":null,"openalex_id":"W2169555996","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Optimization and Variational Analysis"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1093/nsr/nwac035","kind":"article","title":"Neuro-symbolic approaches in artificial intelligence","authors":["Pascal Hitzler","Aaron Eberhart","Monireh Ebrahimi","Md Kamruzzaman Sarker","Lu Zhou"],"author_count":5,"year":2022,"published":"2022-02-23","venue":"National Science Review","cited_by":167,"doi":"10.1093/nsr/nwac035","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1093/nsr/nwac035","archive_url":"https://space-kitty.com/papers/doi:10.1093/nsr/nwac035","updated":null,"pmid":"35673530","pmcid":null,"openalex_id":"W4213454368","pdf_url":"https://academic.oup.com/nsr/article-pdf/9/6/nwac035/43952953/nwac035.pdf","abstract":"Neuro-symbolic artificial intelligence refers to a field of research and applications that combines machine learning methods based on artificial neural networks, such as deep learning, with symbolic approaches to computing and artificial intelligence (AI), as can be found for example in the AI subfield of knowledge representation and reasoning. Neuro-symbolic AI has a long history; however, it remained a rather niche topic until recently, when landmark advances in machine learning—prompted by deep learning—caused a significant rise in interest and research activity in combining neural and symbolic methods. In this overview, we provide a rough guide to key research directions, and literature pointers for anybody interested in learning more about the field. Neuro-symbolic artificial intelligence can be defined as the subfield of artificial intelligence (AI) that combines neural and symbolic approaches. By neural we mean approaches based on artificial neural networks—sometimes called connectionist or subsymbolic approaches—and in particular this includes deep learning, which has provided very significant breakthrough results in the recent decade, and is fueling the current general interest in AI. By symbolic we mean approaches that rely on the explicit representation of knowledge using formal languages—including formal logic—and the manipulation of language items (‘symbols’) by algorithms to achieve a goal. Mostly, neuro-symbolic AI utilizes formal logic as studied in the knowledge representation and reasoning subfield of AI, but the lines blur, and tasks such as general term rewriting or planning, that may not be framed explicitly in formal logic, bear significant similarities and should reasonably be included. Two major reasons are usually brought forth to motivate the study of neuro-symbolic integration. The first one comes from the field of cognitive science, a highly interdisciplinary field that studies the human mind. In that context, we can understand artificial neural networks as an abstraction of the physical workings of the brain, while we can understand formal logic as an abstraction of what we perceive, through introspection, when contemplating explicit cognitive reasoning. In order to advance the understanding of the human mind, it therefore appears to be a natural question to ask how these two abstractions can be related or even unified, or how symbol manipulation can arise from a neural substrate [1]. The second reason is tied to the field of AI and is based on the observation that neural and symbolic approaches to AI complement each other with respect to their strengths and weaknesses. For example, deep learning systems are trainable from raw data and are robust against outliers or errors in the base data, while symbolic systems are brittle with respect to outliers and data errors, and are far less trainable. Symbolic systems, on the other hand, can make explicit use of expert knowledge, and are to a high extent self-explanatory, as their algorithms can be inspected and understood in detail by a human, while neural learning systems cannot readily take advantage of available coded expert knowledge, and are black boxes that make understanding their decision making processes very hard. It is therefore natural to ask how neural and symbolic approaches can be combined or even unified in order to overcome the weaknesses of either approach. Traditionally, in neuro-symbolic AI research, emphasis is on either incorporating symbolic abilities in a neural approach, or coupling neural and symbolic components such that they seamlessly interact [2]. Research in neuro-symbolic AI has a very long tradition, and we refer the interested reader to overview works such as Refs [1,3] that were written before the most recent developments. Indeed, neuro-symbolic AI has seen a significant increase in activity and research output in recent years, together with an apparent shift in emphasis, as discussed in Ref. [2]. Below, we identify what we believe are the main general research directions the field is currently pursuing. It is of course impossible to give credit to all nuances or all important recent contributions in such a brief overview, but we believe that our literature pointers provide excellent starting points for a deeper engagement with neuro-symbolic AI topics. Like in so many other respects, deep learning has had a major impact on neuro-symbolic AI in recent years. This appears to manifest, on the one hand, in an almost exclusive emphasis on deep learning approaches as the neural substrate, while previous neuro-symbolic AI research often deviated from standard artificial neural network architectures [2]. On the other hand, the deep learning context appears to have led to a renewed realization of the importance of neuro-symbolic AI research, and consequently a significant increase in research papers, meetings and prominent public appearances of the topic [2], as well as discussion of the topic in public media [4]. This increase in activity is probably primarily due to the fact that advances in deep learning now make it possible to address challenge problems in neuro-symbolic AI that were quite out of reach before the advent of deep learning, thus adding to its attractivity for research and applications. However, we may also be seeing indications or a realization that pure deep-learning-based methods are likely going to be insufficient for certain types of problems that are now being investigated from a neuro-symbolic perspective. While we cannot give the whole neuro-symbolic AI field due recognition in a brief overview, we have attempted to identify the major current research directions based on our survey of recent literature, and we present them below. Literature references within this text are limited to general overview articles, but a supplementary online document referenced at the end contains references to concrete examples from the recent literature. Examples for historic overview works that provide a perspective on the field, including cognitive science aspects, prior to the recent acceleration in activity, are Refs [1,3]. Very recent overview articles include Refs [2,5–9]. And recent article collections are, e.g. Refs [10–12]. Solving symbolic problems with deep learning. In this line of effort, deep learning systems are trained to solve problems such as term rewriting, planning, elementary algebra, logical deduction or abduction or rule learning. These problems are known to often require sophisticated and non-trivial symbolic algorithms. Attempting these hard but well-understood problems using deep learning adds to the general understanding of the capabilities and limits of deep learning. It also provides deep learning modules that are potentially faster (after training) and more robust to data imperfections than their symbolic counterparts. Using symbolic knowledge bases and expressive metadata to improve deep learning systems. Metadata that augments network input is increasingly being used to improve deep learning system performances, e.g. for conversational agents. Metadata are a form of formally represented background knowledge, for example a knowledge base, a knowledge graph or other structured background knowledge, that adds further information or context to the data or system. In its simplest form, metadata can consist just of keywords, but they can also take the form of sizeable logical background theories. Neuro-symbolic lines of work include the use of knowledge graphs to improve zero-shot learning. Background knowledge can also be used to improve out-of-sample generalizability, or to ensure safety guarantees in neural control systems. Other work utilizes structured background knowledge for improving coherence and consistency in neural sequence models. In a similar manner, natural language fact statements can be used as background knowledge for deep-learning-based conversation agents—strictly speaking, we may not call this approach neuro-symbolic as it uses natural language rather than structured metadata, but of course work like this, using natural language, can be closely related. Explainability through background knowledge. How to explain the input-output behavior, or even inner activation states, of deep learning networks is a highly important line of investigation, as the black-box character of existing systems hides system biases and generally fails to provide a rationale for decisions. Recently, awareness is growing that explanations should not only rely on raw system inputs but should reflect background knowledge. Complex problem solving through coupling of deep learning and symbolic components. Coupled neuro-symbolic systems are increasingly used to solve complex problems such as game playing or scene, word, sentence interpretation. Coupled systems can also be used to make deep learning more sample efficient, for example by using symbolic planning to arrive at (among other things) more data-efficient reinforcement learning, or the use of coupling for vision and language understanding that also results in a more data- and memory-efficient approach. In a different line of work, logic tensor networks in particular have been designed to capture logical background knowledge to improve image interpretation, and neural theorem provers can provide natural language reasoning by also taking knowledge bases into account. Coupling may be through different methods, including the calling of deep learning systems within a symbolic algorithm, or the acquisition of symbolic rules during training. Very tight coupling can be achieved for example by means of Markov logics. Summarizing, neuro-symbolic artificial intelligence is an emerging subfield of AI that promises to favorably combine knowledge representation and deep learning in order to improve deep learning and to explain outputs of deep-learning-based systems. Neuro-symbolic approaches carry the promise that they will be useful for addressing complex AI problems that cannot be solved by purely symbolic or neural means. We have laid out some of the most important currently investigated research directions, and provided literature pointers suitable as entry points to an in-depth study of the current state of the art. This work was partially supported by the National Science Foundation (NSF) under award OIA-2033521—KnowWhereGraph: Enriching and Linking Cross-Domain Knowledge Graphs using Spatially-Explicit AI Technologies. Conflict of interest statement. None declared.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Logic, Reasoning, and Knowledge"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Neuro-symbolic approaches in artificial intelligence\" by Pascal Hitzler, Aaron Eberhart, Monireh Ebrahimi et al., https://doi.org/10.1093/nsr/nwac035, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"doi:10.1109/tcyb.2013.2276043","kind":"article","title":"Adaptive Neural Network Output Feedback Control for Stochastic Nonlinear Systems With Unknown Dead-Zone and Unmodeled Dynamics","authors":["Shaocheng Tong","Tong Wang","Yongming Li","Huaguang Zhang"],"author_count":4,"year":2014,"published":"2014-05-13","venue":"IEEE Transactions on Cybernetics","cited_by":204,"doi":"10.1109/tcyb.2013.2276043","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tcyb.2013.2276043","archive_url":"https://space-kitty.com/papers/doi:10.1109/tcyb.2013.2276043","updated":null,"pmid":"24013830","pmcid":null,"openalex_id":"W2024677911","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Adaptive Control of Nonlinear Systems"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"openalex:W2963519230","kind":"article","title":"Convergence Analysis of Two-layer Neural Networks with ReLU Activation","authors":["Yuanzhi Li","Yuan Yang"],"author_count":2,"year":2017,"published":"2017-05-01","venue":"neural information processing systems","cited_by":192,"doi":null,"arxiv_id":null,"nct_id":null,"source_url":"https://papers.nips.cc/paper/6662-convergence-analysis-of-two-layer-neural-networks-with-relu-activation.pdf","archive_url":"https://space-kitty.com/papers/openalex:W2963519230","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W2963519230","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Stochastic Gradient Optimization Techniques"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1109/tii.2017.2684820","kind":"article","title":"Neural Network Learning Adaptive Robust Control of an Industrial Linear Motor-Driven Stage With Disturbance Rejection Ability","authors":["Ze Wang","Chuxiong Hu","Yu Zhu","Suqin He","Kaiming Yang","Ming Zhang"],"author_count":6,"year":2017,"published":"2017-03-20","venue":"IEEE Transactions on Industrial Informatics","cited_by":177,"doi":"10.1109/tii.2017.2684820","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tii.2017.2684820","archive_url":"https://space-kitty.com/papers/doi:10.1109/tii.2017.2684820","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W2598353255","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Adaptive Control of Nonlinear Systems"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:1804.10306","kind":"article","title":"Universal Approximations of Invariant Maps by Neural Networks","authors":["Dmitry Yarotsky"],"author_count":1,"year":2021,"published":"2021-04-30","venue":"Constructive Approximation","cited_by":139,"doi":"10.1007/s00365-021-09546-1","arxiv_id":"1804.10306","nct_id":null,"source_url":"https://doi.org/10.1007/s00365-021-09546-1","archive_url":"https://space-kitty.com/papers/arxiv:1804.10306","updated":"2018-04-26","pmid":null,"pmcid":null,"openalex_id":"W2799020113","pdf_url":"https://arxiv.org/pdf/1804.10306","abstract":"We describe generalizations of the universal approximation theorem for neural networks to maps invariant or equivariant with respect to linear representations of groups. Our goal is to establish network-like computational models that are both invariant/equivariant and provably complete in the sense of their ability to approximate any continuous invariant/equivariant map. Our contribution is three-fold. First, in the general case of compact groups we propose a construction of a complete invariant/equivariant network using an intermediate polynomial layer. We invoke classical theorems of Hilbert and Weyl to justify and simplify this construction; in particular, we describe an explicit complete ansatz for approximation of permutation-invariant maps. Second, we consider groups of translations and prove several versions of the universal approximation theorem for convolutional networks in the limit of continuous signals on euclidean spaces. Finally, we consider 2D signal transformations equivariant with respect to the group SE(2) of rigid euclidean motions. In this case we introduce the \"charge--conserving convnet\" -- a convnet-like computational model based on the decomposition of the feature space into isotypic representations of SO(2). We prove this model to be a universal approximator for continuous SE(2)--equivariant signal transformations.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Neural Networks and Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1109/tsmc.2016.2557223","kind":"article","title":"Adaptive Neural Network Control of Biped Robots","authors":["Changyin Sun","Wei He","Weiliang Ge","Cheng Chang"],"author_count":4,"year":2016,"published":"2016-01-01","venue":"IEEE Transactions on Systems Man and Cybernetics Systems","cited_by":151,"doi":"10.1109/tsmc.2016.2557223","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tsmc.2016.2557223","archive_url":"https://space-kitty.com/papers/doi:10.1109/tsmc.2016.2557223","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W2390714945","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Robotic Locomotion and Control"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:1905.04943","kind":"preprint","title":"Universal Invariant and Equivariant Graph Neural Networks","authors":["Nicolas Keriven","Gabriel Peyré A"],"author_count":2,"year":2019,"published":"2019-05-13","venue":null,"cited_by":146,"doi":null,"arxiv_id":"1905.04943","nct_id":null,"source_url":"http://arxiv.org/abs/1905.04943","archive_url":"https://space-kitty.com/papers/arxiv:1905.04943","updated":"2019-10-24","pmid":null,"pmcid":null,"openalex_id":"W2943931381","pdf_url":"https://arxiv.org/pdf/1905.04943","abstract":"Graph Neural Networks (GNN) come in many flavors, but should always be either invariant (permutation of the nodes of the input graph does not affect the output) or equivariant (permutation of the input permutes the output). In this paper, we consider a specific class of invariant and equivariant networks, for which we prove new universality theorems. More precisely, we consider networks with a single hidden layer, obtained by summing channels formed by applying an equivariant linear operator, a pointwise non-linearity and either an invariant or equivariant linear operator. Recently, Maron et al. (2019) showed that by allowing higher-order tensorization inside the network, universal invariant GNNs can be obtained. As a first contribution, we propose an alternative proof of this result, which relies on the Stone-Weierstrass theorem for algebra of real-valued functions. Our main contribution is then an extension of this result to the equivariant case, which appears in many practical applications but has been less studied from a theoretical point of view. The proof relies on a new generalized Stone-Weierstrass theorem for algebra of equivariant functions, which is of independent interest. Finally, unlike many previous settings that consider a fixed number of nodes, our results show that a GNN defined by a single set of parameters can approximate uniformly well a function defined on graphs of varying size.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":false,"categories":["Advanced Graph Neural Networks"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1109/tnnls.2022.3143494","kind":"article","title":"Neural-Network-Based Finite-Time Bipartite Containment Control for Fractional-Order Multi-Agent Systems","authors":["Yang Liu","Huaguang Zhang","Zhan Shi","Zhiyun Gao"],"author_count":4,"year":2022,"published":"2022-01-31","venue":"IEEE Transactions on Neural Networks and Learning Systems","cited_by":131,"doi":"10.1109/tnnls.2022.3143494","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tnnls.2022.3143494","archive_url":"https://space-kitty.com/papers/doi:10.1109/tnnls.2022.3143494","updated":null,"pmid":"35100125","pmcid":null,"openalex_id":"W4210310925","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Distributed Control Multi-Agent Systems"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1016/j.fss.2012.01.005","kind":"article","title":"Exponential and almost sure exponential stability of stochastic fuzzy delayed Cohen–Grossberg neural networks","authors":["Quanxin Zhu","Xiaodi Li"],"author_count":2,"year":2012,"published":"2012-01-20","venue":"Fuzzy Sets and Systems","cited_by":150,"doi":"10.1016/j.fss.2012.01.005","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1016/j.fss.2012.01.005","archive_url":"https://space-kitty.com/papers/doi:10.1016/j.fss.2012.01.005","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W1992652478","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Neural Networks Stability and Synchronization"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2006.07579","kind":"article","title":"Stability Analysis Using Quadratic Constraints for Systems With Neural Network Controllers","authors":["He Yin","Peter J. Seiler","Murat Arcak"],"author_count":3,"year":2021,"published":"2021-03-29","venue":"IEEE Transactions on Automatic Control","cited_by":133,"doi":"10.1109/tac.2021.3069388","arxiv_id":"2006.07579","nct_id":null,"source_url":"https://doi.org/10.1109/tac.2021.3069388","archive_url":"https://space-kitty.com/papers/arxiv:2006.07579","updated":"2021-01-26","pmid":null,"pmcid":null,"openalex_id":"W3142744390","pdf_url":"https://arxiv.org/pdf/2006.07579","abstract":"A method is presented to analyze the stability of feedback systems with neural network controllers. Two stability theorems are given to prove asymptotic stability and to compute an ellipsoidal inner-approximation to the region of attraction (ROA). The first theorem addresses linear time-invariant systems, and merges Lyapunov theory with local (sector) quadratic constraints to bound the nonlinear activation functions in the neural network. The second theorem allows the system to include perturbations such as unmodeled dynamics, slope-restricted nonlinearities, and time delay, using integral quadratic constraint (IQCs) to capture their input/output behavior. This in turn allows for off-by-one IQCs to refine the description of activation functions by capturing their slope restrictions. Both results rely on semidefinite programming to approximate the ROA. The method is illustrated on systems with neural networks trained to stabilize a nonlinear inverted pendulum as well as vehicle lateral dynamics with actuator uncertainty.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Stability and Control of Uncertain Systems"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1109/tnnls.2015.2462127","kind":"article","title":"Adaptive Neural Control for a Class of Pure-Feedback Nonlinear Systems via Dynamic Surface Technique","authors":["Zongcheng Liu","Xinmin Dong","Jianping Xue","Hongbo Li","Yong Chen"],"author_count":5,"year":2015,"published":"2015-08-13","venue":"IEEE Transactions on Neural Networks and Learning Systems","cited_by":138,"doi":"10.1109/tnnls.2015.2462127","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tnnls.2015.2462127","archive_url":"https://space-kitty.com/papers/doi:10.1109/tnnls.2015.2462127","updated":null,"pmid":"26277010","pmcid":null,"openalex_id":"W2411555204","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Adaptive Control of Nonlinear Systems"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1109/tnnls.2024.3386881","kind":"article","title":"Pinning-Based Neural Control for Multiagent Systems With Self-Regulation Intermediate Event-Triggered Method","authors":["Hongru Ren","Zeyi Liu","Hongjing Liang","Hongyi Li"],"author_count":4,"year":2024,"published":"2024-04-22","venue":"IEEE Transactions on Neural Networks and Learning Systems","cited_by":110,"doi":"10.1109/tnnls.2024.3386881","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tnnls.2024.3386881","archive_url":"https://space-kitty.com/papers/doi:10.1109/tnnls.2024.3386881","updated":null,"pmid":"38648124","pmcid":null,"openalex_id":"W4394994803","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Distributed Control Multi-Agent Systems"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:1908.05202","kind":"article","title":"Embedding and approximation theorems for echo state networks","authors":["Allen Hart","James Hook","Jonathan Dawes"],"author_count":3,"year":2020,"published":"2020-05-16","venue":"Neural Networks","cited_by":126,"doi":"10.1016/j.neunet.2020.05.013","arxiv_id":"1908.05202","nct_id":null,"source_url":"https://doi.org/10.1016/j.neunet.2020.05.013","archive_url":"https://space-kitty.com/papers/arxiv:1908.05202","updated":"2020-05-18","pmid":"32447266","pmcid":null,"openalex_id":"W3025101483","pdf_url":"https://purehost.bath.ac.uk/ws/files/205890952/HartHookDawes_ESN_embeddings_preprint_JAN2020.pdf","abstract":"Echo State Networks (ESNs) are a class of single layer recurrent neural networks that have enjoyed recent attention. In this paper we prove that a suitable ESN, trained on a series of measurements of an invertible dynamical system, induces a C1 map from the dynamical system's phase space to the ESN's reservoir space. We call this the Echo State Map. We then prove that the Echo State Map is generically an embedding with positive probability. Under additional mild assumptions, we further conjecture that the Echo State Map is almost surely an embedding. For sufficiently large, and specially structured, but still randomly generated ESNs, we prove that there exists a linear readout layer that allows the ESN to predict the next observation of a dynamical system arbitrarily well. Consequently, if the dynamical system under observation is structurally stable then the trained ESN will exhibit dynamics that are topologically conjugate to the future behaviour of the observed dynamical system. Our theoretical results connect the theory of ESNs to the delay-embedding literature for dynamical systems, and are supported by numerical evidence from simulations of the traditional Lorenz equations. The simulations confirm that, from a one dimensional observation function, an ESN can accurately infer a range of geometric and topological features of the dynamics such as the eigenvalues of equilibrium points, Lyapunov exponents and homology groups.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by-nc-nd","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Neural Networks and Reservoir Computing"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1109/tcyb.2021.3091580","kind":"article","title":"Robust Adaptive Neural Control for Wing-Sail-Assisted Vehicle via the Multiport Event-Triggered Approach","authors":["Guoqing Zhang","Jiqiang Li","Xu Jin","Cheng Liu"],"author_count":4,"year":2021,"published":"2021-08-24","venue":"IEEE Transactions on Cybernetics","cited_by":124,"doi":"10.1109/tcyb.2021.3091580","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tcyb.2021.3091580","archive_url":"https://space-kitty.com/papers/doi:10.1109/tcyb.2021.3091580","updated":null,"pmid":"34260374","pmcid":null,"openalex_id":"W3178576965","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Adaptive Control of Nonlinear Systems"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1007/s00521-014-1744-4","kind":"article","title":"Solving time-varying quadratic programs based on finite-time Zhang neural networks and their application to robot tracking","authors":["Peng Miao","Yanjun Shen","Yuehua Huang","Yan‐Wu Wang"],"author_count":4,"year":2014,"published":"2014-10-17","venue":"Neural Computing and Applications","cited_by":129,"doi":"10.1007/s00521-014-1744-4","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1007/s00521-014-1744-4","archive_url":"https://space-kitty.com/papers/doi:10.1007/s00521-014-1744-4","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W1983152073","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"retracted":false,"peer_reviewed":null,"categories":["Neural Networks and Applications"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."}],"next":"https://space-kitty.com/api/v1/topics/ai-mathematics/papers?sort=top&limit=25&offset=25"}