Inter national J our nal of Inf ormatics and Communication T echnology (IJ-ICT) V ol. 15, No. 4, December 2026, pp. 1902 ∼ 1920 ISSN: 2252-8776, DOI: 10.11591/ijict.v15i4.pp1902-1920 ❒ 1902 FusionGait-transf ormer: multimodal transf ormer -based framew ork f or r ob ust gait r ecognition and gait-condition classication Kiran K umar S V N Madupu 1 , J agadeeshwar Madiraju 1 , Nar en K odali 2 , Soma Sekhar Go viraboyina 3 1 Chaitan ya Deemed to be Uni v ersity , Himayatnag ar , India 2 Geor ge Mason Uni v ersity , Uni v ersity Dri v e, F airf ax, V ir ginia, United States 3 Department of Computer Science and Engineering, Anurag Uni v ersity , Hyderabad, India Article Inf o Article history: Recei v ed Feb 4, 2026 Re vised Aug 11, 2026 Accepted Aug 28, 2026 K eyw ords: Cross-attention fusion Domain adaptation Edge deplo yment Gait identication Gait-condition classication Multimodal g ait analysis W earable IMU ABSTRA CT Gait analysis benets from both full-body motion conte xt and accurate tempo- ral dynamics. V ision-only approaches de grade under occlusion and vie wpoint change, while inertial-only approaches miss full body conte xt. FusionGait- transformer is a multimodal frame w ork that fuses pose and silhouette repre- sentations with inertial time series using transformer self-attention and bidirec- tional cross-attention. A per -modality reliability g ate weights each stream by its instantaneous signal quality , so de gradation in one modality shifts the fused representation to w ard the other . Mask ed multimodal pretraining and domain- adv ersarial training impro v e stability under domain shift. Ev aluation uses the CMU motion capture dataset under a subject-independent protocol, with iner - tial signals generated analytically from the same trajectories. The frame w ork reaches Rank-1 identication accurac y of 98.9% with equal error rate (EER) of 1.8%, and g ait-condition classication F1-score of 95.7% with area under the R OC curv e (A UC-R OC) of 97.6%. Reliability g ating is t he lar gest single con- trib utor t o these gures. Deplo yment is characterized on w orkstation, edge, and g ate w ay hardw are, and transmitting embeddings rather than video reduces the uplink requirement by more than four orders of magnitude, establishing feasi- bility for wearable healthcare settings. Reported results describe performance under controlled benchmark conditions. Clinical diagnostic claims remain con- ditional on v alidation with real wearable sensors and clinically labeled cohorts. This is an open access article under the CC BY -SA license . Corresponding A uthor: Kiran K umar S V N Madupu Chaitan ya Deemed to be Uni v ersity Himayatnag ar , T elang ana-500075, India Email: kirankumarsvnmadupu@gmail.com 1. INTR ODUCTION Gait analysis supports detection of health conditions through quantitati v e study of w alking patterns [1], [2]. Changes in these patterns can indicate neurological or muscul osk eletal disorders and pro vide early mark ers for interv ention [2]. T raditional clinical assessments rely on basic speed or balance tests, which pro- duce inconsistent and limited results [3]-[6]. Specialized motion-capture labs deli v er detailed measurements b ut remain costly and impracti cal for routine clinical use [1], [7]. A simpler and accessible approach for e v ery- day clinical use remains lacking. J ournal homepage: http://ijict.iaescor e .com Evaluation Warning : The document was created with Spire.PDF for Python.
Int J Inf & Commun T echnol ISSN: 2252-8776 ❒ 1903 Single-method g ait systems each ha v e clear dra wbacks outside controlled en vironments. Lab- b a sed motion-capture de vices of fer precise measurement b ut are di f cult to use re gularly because of equipment and technical demands [7]. Mark erless video analysis assesses g ait without intrus i v e de vices, yet it typically needs se v eral cameras and precise setup, and it de grades under camera blockage or unusual body mo v ement [8]-[10]. Pri v ac y concerns and data management further limi t its adoption [11]. W earable sensors such as IMUs pro vide continuous tracking outside the lab b ut capture limited mo v ement data from a fe w body se gments [12]-[14]. The y often cannot identify conditions in v olving interaction between limbs or with en vironmental obstacles [15]-[20]. Each single-modality system sacrices accurac y , usability , or practicality . Gait assessment reaches patients through an information and communication technology pipel ine rather than through a model in isolation. W earable de vices already sense mo v ement continuously inside clinical w alking assessments [12], and video adds a second sensing channel that carries identiable data of f the capture site. Where the intermediate processing tier should sit, on the wearable node, a clinical g ate w ay , or a serv er , is therefore part of the design problem rather than an implementation detail. The quest ion is not accurac y alone, b ut whether a g ait representation can be produced, transported, and interpreted within the constraints of a deplo yed healthcare netw ork. Recent studies combine vision and sensor data to o v ercome these indi vidual limits [21], [22]. V ideo supplies full-body mo v ement and en vironmental conte xt, while wearable sensors capture rapid limb motion precisely . P airing IMUs with video impro v es assessment of f all-related w alking problems by including conte xt such as obstacles or terrain [19]. T ransformers manage long sequences and model relationships across data streams, which suits combined g ait data [23]. The dual-stream vision transformer of Li et al. [22] reached about 97% accurac y for clinical g ait analysis from a single lo w-cost RGB camera, and attenti on-based frame w orks adapt to user v ariation in related acti vity recognition [21]. Multimodal transformers in medical applications combine heterogeneous sources to identify conditions [23]. FusionGait-transformer inte grates vision and inert ial signals in a unied sequence model. One st ream encodes visual g ait cues from pose and silhouette representations. A second stream encodes synchronized IMU signals, including linear acceleration and angular v elocity . A transformer fusion module combines both streams using cross-attention to model temporal alignment and cross-modal interaction within a g ait c ycle. The fused embedding supports g ait identication and g ait-condition classication dened by the e v aluation proto- col. Mask ed multimodal pretraining and domain-adv ersarial learning impro v e rob ustness across vie wpoint and sensor v ariation. Prior multimodal g ait studies either combine tw o visual representations, pose with silhouette, without inertial signals, or combine inertial signals with one visual cue, or apply late or shallo w fusion across modalities. No published architecture jointly inte grates pose, silhouette, and inertial signals within a single unied cross- attention T ransformer e v aluated under a subject-independent protocol. Deplo yment is also treated separately: reported e v aluations generally co v er one or tw o sensing streams, and e x ecution cost is rarely reported alongside recognition performance. FusionGait-transformer closes both g aps. K e y contrib utions are summarized as follo ws; (i) A multimodal transformer fusion architecture com- bines pose and silhouette embeddings with IMU embeddings using bidirectional cross-attention and per -modality reliability g ating. (ii) A training strate gy combines mask ed multimodal pretraining with domain-adv ersarial learning to support domain-in v ariant g ait representations. (iii) Clinically moti v ated g ait metrics, including symmetry and re gularity indices, are inte grated with learned embeddings for g ait-condition classication and interpretability . (i v) A benchmar k protocol is dened using subject-independent spli ts and control led domain f actors such as speed, vie wpoint, and sensor placement perturbation. (v) An e x ecution prole is reported across w orkstation, edge, and g ate w ay tiers, together with the bandwidth implication of transmitting embed- dings rather than video, which establishes ICT feasibility alongside accurac y . The remainder of the manuscript is or g anized as follo ws. Section 2 re vie ws related w ork. Section 3 describes data preparation, feature encoding, and the fusion architecture. Section 4 describes the e xperimental protocol and results. Section 5 summarizes ndings and limitations. 2. RELA TED W ORK T ransformer models ha v e impro v ed g ait recognition. Cosma and Radoi presented GaitF ormer , a trans- former o v er g ait sk eleton sequences, reaching 92.5% accurac y on CASIA-B without manual labels [24]. Mog an et al. [25] proposed Gait-V iT , applying V ision T ransformers to silhouett e images and reporting 99 to 100% FusionGait-tr ansformer: multimodal tr ansformer -based fr ame work ... (Kir an K umar S V N Madupu) Evaluation Warning : The document was created with Spire.PDF for Python.
1904 ❒ ISSN: 2252-8776 recognition on common benchmarks. T ransformers e xtend be yond identication. The attention-enhanced net- w ork of Jiao et al. [26], which shares the GaitF ormer name b ut addresses a dif ferent task, inte grates g ait patterns and pressure data to detect early P arkinson’ s disease with 92.3% accurac y . GaitF oreMer by Endo e t al. [27] used self-supervised learning t o predict P arkinson’ s se v erity and impro v ed clinical scoring. These w orks sho w that transformers learn detailed g ait features suitable for identication and clinical use. Multimodal g ait analysis combining visual methods has recei v ed increased attention, because single- modality visual approaches that analyze silhouettes or sk eletons alone de grade under clothing change or v arying vie wpoint. Xu et al. [28] proposed PSGait, o v erlaying parsing-deri v ed body-part sk eletons onto silhouettes and reporting accurac y g ains in less controlled scenarios. Unconstrained recognition has also been approached through dense 3D representations on a dedicated benchmark [29]. The Sk eletonGait++ v ariant mer ges sk eleton maps with silhouettes using att ention, impro ving identication under changing vie ws [30]. Peng et al. [31] lik e wise inte grate sk eletons and sil houettes into a richer feature set. Related parsing-based approaches use learned semantic se gmentation of t he w alking body , in the wild [32] and through human semantic parsing [33]. Cui and Kang [34] proposed a spatial-temporal fusion model combining complementary visual g ait streams, with consistent g ains o v er single-stream baselines. These methods generally outperform single-modality visual approaches. The g ains, ho we v er , remain within the visual modality and do not address sensor -domain artef acts. W earable sensor fusion has also adv anced g ait analysis. IMU sensors placed on body se gments allo w continuous tracking outside lab settings, and machine learning o v er these signals classies abnormal g ait while accounting for joint impairment [35]. Early sensor -based methods combined multiple IMUs with deep con v o- lutional netw orks for g ait recognition [36]. Nguyen et al. [37] applied a spectrogram representation with deep con v olutional netw orks for g ait classication. Uchitomi et al. [38] classied mild P arkinson’ s disease from inertial time-series data, relying on augmentation to of fset limited sample size. Shari-Renani et al. [39] de v el- oped BioMA T , a transformer inte grating wear able sensor signals to predict joint kinematics across acti vities, surpassing CNN-LSTM methods. These models usually tar get specic g ait tasks and rarely inte grate sensor and vision data fully . A smal ler body of w ork treats g ait anal ysis as an ICT problem rather than a modelling problem alone. T errain and en vironment classication from wearable streams has been used to conte xtualize f all risk in daily life [11]. Continuous wearable platforms ha v e been deplo yed for multidomain diagnostic and interv ention w orko ws, placing the model inside a care pathw ay rather than an of ine pipeline [14]. V ideo capture in the same setting introduces go v ernance requirements that shape what data may lea v e the point of capture [20]. Systems designed for de graded operating conditions sho w that fusion can be or g anized around deplo yment constraints as well as accurac y [19]. Ev aluations in this group generally co v er one or tw o sensing streams, and e x ecution cost is rarely reported alongside recognition performance. Self-supervised learning and domain adaptation address limited g ait datasets and impro v e general- ization. GaitF oreMer used self-supervised forecasting of human mo v ement to impro v e clinical impairment prediction [27], and GaitF ormer used weak supervision to impro v e identication without manual labeling [24]. Unsupervised domain adaptation, such as the ne-grained approach of Ma et al. [40], allo ws g ait models to adapt across vie wing angles and clothing types without labeled data. Inertial-sensor frame w orks for patholog- ical g ait recognition report generalization to patient data [41], real-time neural classication has been demon- strated in li ving en vironments [42], and domain adaptation for inertial acti vity recognition has been surv e yed [43]. These strate gies impro v e stability b ut need v alidation in clinical conte xts. Most transformer and multimodal g ait studies focus on identication or broad impairment, rarely addressing detailed clinical diagnostics. The 3DGK model of Lan et al. [44] combined 3D g ait data with deep learni ng, classifying pediatric neurological conditions and estimating strok e onset times (A UC 0.77 to 0.99). That approach depends on controlled lab en vironments with mark er -based systems. TDPT -GT by Iseki et al. [45] used smartphone-based mark erless motion capture, classi fying neurological g ait disorders with moderate accurac y (A UC 0.719), g aining portability b ut lacking diagnostic precision. These studies tar get either controlled laboratory accurac y or port able b ut less precise assessment, lea ving a g ap for multimodal inte gration in detailed clinical diagnostics. Pre vious g ait studies share se v eral limitations. Man y analyze one modality only , o v erlooking cross- modal interactions such as subtle limb mo v ements visibl e in one stream b ut unclear in another [24], [25], [45]. Clinical diagnostics often remain coarse, distinguishing health y from pathological g ait rather than specic dis- orders [45]. Generalization across patient groups or en vironments remains problematic without adaptation [40], [41]. Clinical g ait datasets remain small, placing weight on augmentation and pretraining, since representation Int J Inf & Commun T echnol, V ol. 15, No. 4, December 2026: 1902–1920 Evaluation Warning : The document was created with Spire.PDF for Python.
Int J Inf & Commun T echnol ISSN: 2252-8776 ❒ 1905 quality scales with the v olume of data a v ailabl e [46]. Inte gration of video and sensor data into one approach has ra rely been achie v ed, since e xisting systems either isolate modalities or combine them supercially [45]. These issues limit adoption of AI-based g ait systems in clinical practice. FusionGait-transformer addresses these limitations by combining synchronized visual g ait repre sen- tations and inertial signals in a single transformer model. Cross-attention connects temporally aligned cues across modalities and supports learning of complementary features. Per -modali ty reliability g ating weights each stream by its instantaneous s ignal quality , so de gradation in one modality shifts the fused representation to w ard the other . The frame w ork also computes clinically moti v ated metrics, including symmetry and re gu- larity indices, and combines these with learned embeddings for g ait-condition classication under the dened label protocol. Mask ed pretraining and domain-adv ersarial learning impro v e generalization across vie wpoint and sensor v ariation. Ex ecution cost is reported alongside accurac y so that the design can be assessed as a deplo yable component of a healthcare ICT pipeline. 3. METHOD FusionGait-transformer is described in pipeline order: data acquisition and preprocessing (Section 3.2.), modality-specic fea ture encoding (section 3.2), cross-attention fusion with modality reli ability g ating (section 3.3), and clinically moti v ated metric computation with classication-head inte gration (section 3.5). The joint training objecti v e, mask ed multimodal pretraining, and domain-adv ersarial training are detail ed in section 4 together with the e xperimental protocol. T w o data acquisition settings are dened. The deplo yment setting uses RGB video and wearable IMU sensors. The e v aluation setting uses motion-capture trajectories with analytically generated inertial signals and projected pose sequences deri v ed from the same trajectories. Notation used throughout the manuscript is summarized belo w . Notation – t : time inde x in a g ait sequence. – J : number of joints in the pose representation. – I t : silhouette image at time t . – P 2D t : 2D joint coordinates at time t . – P 3D t : 3D joint coordinates at time t . – c t : hip-center coordinate used for pose normalization. – ˆ P 2D t : normalized 2D pose coordinates. – R 2D t : relati v e 2D joint coordinates. – θ t : joint angle feature v ector . – τ t : normalized g ait-phase timing. – A ( t ) : triaxial acceleration at time t . – Ω( t ) : triaxial angular v elocity at time t . – q ( t ) : unit quaternion orientation at time t . – h V t , h S t : vision and sensor embeddings at time t . – H t : fused embedding at time t . – P E t : positional encoding at time t . – Q , K , Z : query , k e y , and v alue projections in attention. – d k : attention k e y dimension for scaling. – q g : learnable global query v ector used for pooling. – α t : pooling weight for time t . – g : global g ait embedding. – d = d mo del = 256 : dimension of the global g ait embedding. – GSI , CV , SR : symmetry , cadence v ariability , and stride re gularity metrics. – c : clinically moti v ated metric v ector . – z : combined v ector for g ait-condition classication. – L id , L cls , L dom : identity , g ait-condition, and domain-adv ersarial losses. – λ cls , λ dom : loss weights. FusionGait-tr ansformer: multimodal tr ansformer -based fr ame work ... (Kir an K umar S V N Madupu) Evaluation Warning : The document was created with Spire.PDF for Python.
1906 ❒ ISSN: 2252-8776 – d V , d W , d mo del : vision encoder , sensor encoder , and fusion model dimensions. – D V , D W : dimensions of the concatenated per -frame vision and wearable feature v ectors. – A (V → W) t , A (W → V) t : bidirectional cross-attention weight matrices. – g m ( t ) : per -modality reliability g ate at time t . – T a , T ω : Gauss-Mark o v bias time constants of the virtual IMU model. – T s : a v erage stride interv al used for autocorrelation. 3.1. FusionGait-transf ormer ar chitectur e FusionGait-transformer uses a modular multimodal architecture with separate vision and sensor en- coders fol lo wed by a transformer fus ion module. The modular design captures modality-s pecic dynamics before cross-modal inte gration. Each encoder e xtracts discriminati v e temporal and spatial features, which are then fused within the transformer to produce uni ed representations for g ait identication and g ait-condition classication. Encoder design rationale: encoders are selected on three criteria: representational adequac y for the modality , st ability of pret rained feat ures under a limited-s ubject be n c hmark, and deterministic inference cost suitable for edge or g ate w ay e x ecution in a wearable healthcare deplo yment. ResNet-50 is used for silhouette encoding because its depth-to-cost ratio is f a v ourable at silhouette resolution and its ImageNet features transfer stably; deeper alternati v es (ResNet-101, V iT -B/16) raise parameter count without measurable Rank-1 g ain on this benchmark. HRNet-W48 is used for pose estimation because it maintains high-resolution branches throughout the netw ork, which impro v es localization at the distal limb joints carrying most g ait signal; encoder - decoder alternati v es lose that resolution at the bottleneck. A temporal con v olutional netw ork is preferred o v er recurrent or transformer encoders on the inertial stream because its x ed recepti v e eld yields constant per - windo w latenc y and a parameter count tw o orders of magnitude belo w the vision encoder , which matters when the sensor branch e x ecutes on a wearable node rather than a serv er . V ision encoder: the vision encoder uses a pretrained ResNet-50 ne-tuned on g ait silhouette sequences together with joint pose embeddings e xtracted by HRNet-W48. Each video frame t is encoded into a latent v ector h V t , as in (1). h V t = ResNet50 I t , R 2D t , θ t (1) where I t is the g ait silhouette image, R 2D t denotes relati v e joint coordinates, and θ t represents joint angles. The encoder outputs a sequence of temporal embeddings capturing dynamic g ait visual patterns. Sensor encoder: the sensor encoder is a temporal con v olutional netw ork (tCNN) for IMU ti me-series feature e xtraction. Gi v en angular v elocity v ectors Ω f ( t ) and g ait phase timings τ t , it generates frame-wise latent sensor embeddings h S t , as in (2). h S t = tCNN(Ω f ( t ) , τ t ) (2) This encoder e xtracts sequential kinematic patterns inherent in IMU signals, gi ving an accurate temporal rep- resentation of lo wer -limb g ait dynamics. T ransformer -based fusion module: the fusion module performs multimodal inte gration via multi-head self-attention with positional encodings. V ision and sensor encoder outputs are concatenated per time step as H t = [ h V t ; h S t ] . Positional encodings P E t represent relati v e position within g ait c ycles, as in (3). H ′ t = H t + P E t (3) Multi-head self-attention projects H ′ t into query ( Q t ), k e y ( K t ), and v alue ( Z t ) spaces across heads. Attention for head i at timestep t is computed as in (4). A tten tion t,i ( Q t , K t , Z t ) = S oftMax Q t K T t √ d k Z t (4) where d k is the k e y dimension. Cross-attention further renes modality interactions, weighting vision and sensor features according to modality reliability at each timestep. Int J Inf & Commun T echnol, V ol. 15, No. 4, December 2026: 1902–1920 Evaluation Warning : The document was created with Spire.PDF for Python.
Int J Inf & Commun T echnol ISSN: 2252-8776 ❒ 1907 Global feature pooling: post-transformer embeddings are aggre g ated into a compact global g ait s ig- nature by attention-based pooling. A learned global query q g attends across all time steps, yielding weights α t for each embedding H ′′ t , as in (5). α t = exp q T g H ′′ t P t ′ exp q T g H ′′ t ′ , g = X t α t H ′′ t (5) Figure 1. Architecture of FusionGait-transformer The resulting v ector g encodes spatiotemporal information for do wnstream biometric identication and g ait- condition classication. Architectural h yperparameters are congured as follo ws. The vision encoder produces fram e-wise embeddings of dimension d V = 512 from a ResNet-50 backbone concatenated with relati v e joint coordinates and joint a n gl e features. The sensor encoder produces frame-wise embeddings of dimension d W = 128 f rom a four -layer temporal con v olutional netw ork with 64, 128, 128, and 128 channels. The fusion transformer operates at model dimension d mo del = 256 , with 8 attention heads, k e y dimension d k = 32 , and 6 stack ed encoder layers with feed-forw ard e xpansion f actor 4. The global attention pooling layer uses a learnable query of dimension 256. T otal parameter count is approximately 25.3 million, allocated as 23.5 M (vision encoder), 0.4 M (sensor encoder), 1.2 M (fusion transformer with cross-attention and reliability g ates), and 0.2 M (pooling and classication heads). FusionGait-tr ansformer: multimodal tr ansformer -based fr ame work ... (Kir an K umar S V N Madupu) Evaluation Warning : The document was created with Spire.PDF for Python.
1908 ❒ ISSN: 2252-8776 Figure 1 sho ws the frame w ork: multimodal inputs (RGB vision data and wearable IMU data), the modality-specic encoders, and the transformer fusion module with multi-head self-attention and cross-attention. Positional encodings carry g ait-c ycle timing into the fusion stage. The globally pooled embedding feeds tw o parallel branches for identity recognition and g ait-condition classication, the latter combining the g ait signa- ture with the clinically moti v ated metrics of section 3.5. 3.2. Multimodal data collection and pr epr ocessing T w o data acquisition settings are dened to separate intended sensing from the e v aluation instant iation. Deplo yment sensing conguration: an RGB camera records a w alking sequence. A 2D pose es timator e xtracts J joint coordinates per frame, as in (6). W earable IMU sensors on the lo wer limbs record tri axial acceleration and triaxial angular v elocity , as in (7). P 2D t = n x j t , y j t o J j =1 (6) A ( t ) = [ a x ( t ) , a y ( t ) , a z ( t )] T , Ω( t ) = [ ω x ( t ) , ω y ( t ) , ω z ( t )] T (7) Ev aluation setting constructed from motion capture: motion-capture trajectories pro vide 3D joint po- sitions P 3D t . V irtual IMU signals are generated at anatomically dened lo wer -limb l ocations. F or a virtual sensor s with 3D position r s ( t ) , linear acceleration is computed as in (8). A s ( t ) = ¨ r s ( t ) − g + η a ( t ) + b a ( t ) (8) where g is the gra vity v ector , η a ( t ) is zero-mean measurement noise, and b a ( t ) is a slo wly v arying bias term. Angular v elocity is computed from se gment orientation R s ( t ) ∈ SO(3) , as in (9). Sensor -to-se gment misalign- ment is modeled as a x ed rotation R mis , as in (10). Ω s ( t ) = v ee R T s ( t ) ˙ R s ( t ) + η ω ( t ) + b ω ( t ) (9) A meas s ( t ) = R mis A s ( t ) , Ω meas s ( t ) = R mis Ω s ( t ) (10) where η ω ( t ) and b ω ( t ) denote noise and bias. The virtual IMU conguration is dened as follo ws. Se v en anatomical placements are si mulated: pelvis, left and right thighs, left and right shanks, and left and right feet. Each virtual sensor operates at 100 Hz. T riaxial linear acceleration includes Gaussian measurement noise with standard de viation σ a = 0 . 05 m / s 2 and slo wly v arying bias modelled as a rst-order Gauss-Mark o v process with time constant T a = 100 s and steady- state standard de viation 0.02 m / s 2 . T riaxial angular v elocity includes Gaussian noise with σ ω = 0 . 01 rad / s and bias drift with T ω = 100 s and steady-state standard de viation 0.005 rad/s. Bias time constants are denoted T a and T ω , reserving τ t for the g ait-phase v ariable of section 3.3. Sensor -to-se gment m isalignment is sampled per trial as a x ed rotation dra wn from an axis-angle distrib ution with angle uniformly sampled in [0 ◦ , 5 ◦ ] . The analytical model captures noise, slo w bias drift, and x ed misalignment. It does not represent soft-tissue mo- tion, attachment slippage, long-term drift be yond the Gauss-M ark o v assumption, pack et loss, or asynchronous sampling. These omissions bound the scope of the reported results and are quantied by the sensiti vity sweep of section 4. Pose inputs for e v aluation are obtained by projecting 3D joints to 2D using a camera model. F or camera matrix K and e xtrinsics ( R c | t c ) , projection is gi v en by (11). ˜ u j t = K R c p j t + t c , x j t , y j t =   ˜ u j t, 1 ˜ u j t, 3 , ˜ u j t, 2 ˜ u j t, 3 ! (11) Synchronization and g ait-c ycle se gmentation: multimodal sequences are aligned in time. In the de- plo yment setting, timestamps are aligned using a shared trigger . In the e v aluation setting, modalities share a common time base. Gait c ycles are se gmented using heel-strik e e v ents deri v ed from the v ertical acceleration component. Gi v en a windo w T i co v ering one step, heel-strik e time is estimated as in (12). t ∗ i = arg min t ∈ T i a z ( t ) (12) Int J Inf & Commun T echnol, V ol. 15, No. 4, December 2026: 1902–1920 Evaluation Warning : The document was created with Spire.PDF for Python.
Int J Inf & Commun T echnol ISSN: 2252-8776 ❒ 1909 Preprocessing: pose coordinates are normalized to reduce translation and scale v ariation. The hip- center coordinate is dened by (13) and the normalized pose by (14). c t = p hip ,l ( t ) + p hip ,r ( t ) 2 (13) ˆ P 2D t = P 2D t − c t s t (14) Where s t is a scale f actor deri v ed from a stable body se gment length. IMU signals are lo w-pass ltered to suppress high-frequenc y noise. Se gment orientation is represented by a unit quaternion q ( t ) obtained by inte grating angular v elocity , as in (15). ˙ q ( t ) = 1 2 q ( t ) ⊗ Ω q ( t ) , q ( t ) ← q ( t ) ∥ q ( t ) ∥ (15) The resulting synchronized and se gmented sequences ˆ P 2D t and IMU-deri v ed features form the input to the encoders and the fusion transformer . 3.3. F eatur e r epr esentation and encoding V ision features: the vision feature set inte grates tw o representations deri v ed from HRNet-W48 pos e estimates: relati v e joint coordinates and joint angle embeddings. Relati v e joint coordinates R 2D t e xpress each joint position relati v e to the pelvis center , reducing inter -subject anatomical v ariability , as in (16). R 2D t = { ( x j − x p elvis , y j − y p elvis ) } 18 j =1 , j ̸ = p elvis (16) Joint angles θ t for critical joints encode local art iculation. F or joint j , the angle between se gments v a and v b connecting neighbouring joints a , j , b is gi v en by (17). θ j ( t ) = arccos v a · v b ∥ v a ∥ ∥ v b ∥ (17) Silhouette embeddings S sil t are e xtract ed using an ImageNet-pretrained ResNet-50 ne-tuned on silhouette g ait sequences, as in (18). The per -frame vision feature v ector is dened by (19). S sil t = ResNet50( I t ) (18) V t = R 2D t ; θ t ; S sil t (19) W earable features: wearable features capture limb kinematics from IMUs, comprising angular v eloc- ities, stride e v ents, and g ait phase timings. Filtered angular v elocities gi v e instantaneous rotational motion, as in (20). Ω f ( t ) = [ ω x ( t ) , ω y ( t ) , ω z ( t )] T (20) Stride e v ent features are deri v ed from heel-strik e and toe-of f occurrences in the v ertical acceleration signal. Gait phase timings τ t within each c ycle are normalized between 0 (heel-strik e) and 1 (subsequent heel-stri k e), as in (21). The wearable modality then produces the per -frame features of (22). τ t = t − t ∗ i t ∗ i +1 − t ∗ i , t ∈ t ∗ i , t ∗ i +1 (21) W t = [Ω f ( t ); τ t ] (22) Feature inte gration: vision and wearable features are combined at the joint l e v el for each frame, using spatial alignment between modalities. The inte grated multimodal feature v ector M t at frame t is gi v en by (23). M t = [ V t ; W t ] ∈ R D V + D W (23) Where D V and D W denote the dimensionalities of the vision and wearable features. These concatenated v ectors are the inputs consumed by the encoders of section 3.1, which map them to the encoder outputs h V t and h S t of dimension d V and d W . Self-attention of section 3.1 then operates on the concatenated encoder outputs, while the cross-attention of section 3.4 operates on the modality streams before concatenation, so the tw o paths describe successi v e stages of one pipeline rather than alternati v es. FusionGait-tr ansformer: multimodal tr ansformer -based fr ame work ... (Kir an K umar S V N Madupu) Evaluation Warning : The document was created with Spire.PDF for Python.
1910 ❒ ISSN: 2252-8776 3.4. Multimodal featur e fusion (Cr oss-attention mechanism) FusionGait-transformer inte grates multimodal features through a cross-attention mechanism that mod- els interactions between vision and wearable modalities at each timestep of a g ait c ycle. Gi v en vision em- beddings V t ∈ R D V and wearable embeddings W t ∈ R D W at timestamp t , the module generates rened representations by bidirectional attention across modalities. Each modality alternatel y serv es as the query while the complementary modality pro vides k e ys and v alues. F or vision queries attending to wearable features at timestamp t , the projections are gi v en by (24). Q (V → W) t = V t W QV , K (V → W) t = W t W KW , Z (W) t = W t W ZW (24) Where W QV , W KW , W ZW and their re v erse-direction counterparts W QW , W KV , W ZV are learned linear pro- jection matrices. The attention weights A (V → W) t follo w scaled dot-product attention, as in (25), with d k the k e y dimension. The re v erse direction A (W → V) t is dened symmetrically by e xchanging the roles of the tw o modalities, with Q (W → V) t = W t W QW , K (W → V) t = V t W KV , Z (V) t = V t W ZV . A (V → W) t = SoftMax Q (V → W) t K (V → W) t T √ d k (25) Modality rel iability is modelled e xplicitly . A per -modality rel iability g ate g m ( t ) ∈ [0 , 1] is computed for each modality m ∈ { V , W } at time t from the local norm and the local Shannon entrop y of the modality embedding, as in (26). g m ( t ) = σ w T m [ ∥ x m ( t ) ∥ 2 ; H ( x m ( t ))] + b m (26) Where σ is the sigmoid, ∥ x m ( t ) ∥ 2 is the L2 norm of the modality embedding at time t , H ( x m ( t )) is the local Shannon entrop y across feature channels, and w m , b m are learnable parameters. The g ated cross-attention outputs are then computed as in (27) and (28). V fused t = V t + g W ( t ) · A (V → W) t Z (W) t (27) W fused t = W t + g V ( t ) · A (W → V) t Z (V) t (28) When modality m e xhibits lo w local ener gy or high entrop y , signalling noise, occlusion, or sensor artef act, g m ( t ) decreases to w ard zero and reduces the contrib ution of that modality to the fused stream. The g ates are trained jointly with the rest of the netw ork through the identity and g ait-condit ion losses, gi ving an e xplicit and learnable mechanism for per -timestep modality weighting. The mechanism supplies rob ustness through complementarity . Under partial occlusion or adv erse lighting the g ate on the vision stream contracts and IMU-deri v ed features carry the fused representation; under inertial drift or transient sensor a rtef act the re v erse holds. W eighting is therefore recalibrated at the temporal granularity of the g ait c ycle rather than x ed at training time. Measured g ate trajectories and attention mass distrib utions are reported in section 4. 3.5. Clinically moti v ated gait metrics Diagnostic metrics are computed from fused multimodal data (vision-deri v ed 2D joint trajectories and wearable sensor signals) to quantify g ait abnormality . – Gait symmetry inde x (GSI): GSI quanties left-right symmetry . Joint angle trajectories from pose estima- tion and IMU-deri v ed stride e v ent timings serv e as inputs. GSI for joint j is calculated as in (29). GSI j = X L j − X R j 0 . 5 X L j + X R j × 100 (29) Where X L j and X R j represent peak angular displacement or stride duration from the left and right sides. Ag- gre g ation across k e y joints yields a symmetry prole. Int J Inf & Commun T echnol, V ol. 15, No. 4, December 2026: 1902–1920 Evaluation Warning : The document was created with Spire.PDF for Python.
Int J Inf & Commun T echnol ISSN: 2252-8776 ❒ 1911 – Cadence v ariability (CV): CV reects temporal g ait stability . Stride durations ∆ t i determined by consecu- ti v e heel-strik e timings, augmented by frame-based pose se gmentation, gi v e the coef cient of v ariation, as in (30). CV = σ (∆ t i ) µ (∆ t i ) × 100 (30) Where σ (∆ t i ) and µ (∆ t i ) denote the standard de viation and mean of stride duration across g ait c ycles. – Stride re gularity (SR): SR measures consistenc y of repetiti v e stride patterns by autocorrelation of the v erti- cal acceleration signal a z ( t ) e v aluated at the a v erage stride interv al T s , as in (31). SR = R a z ( T s ) = E [ a z ( t ) a z ( t + T s )] p E [ a 2 z ( t )] E [ a 2 z ( t + T s )] (31) Where E is the e xpectation operator , and higher SR indicates more c on s istent stride patterns. These metrics pro vide a diagnostic prole reecting underlying g ait dysfunction. Inte gration into classication head: the metric v ector c = [GSI , CV , SR] T is concatenated with the global g ait embedding g ∈ R d to form (32). A classication head implemented as a fully connected netw ork with SoftMax output predicts the g ait-condition label y dened in the e v aluation protocol, as in (33). z = [ g ; c ] ∈ R d +3 (32) p ( y | z ) = SoftMax( W c z + b ) (33) Joint training with the identity recognition head uses the composite objecti v e of (34). L = L id + λ cls L cls (34) Where L id and L cls are cross-entrop y losses for identity recognition and g ait-condition classication, and λ cls balances task contrib utions. The joint objecti v e encourages embeddings that pre serv e identity-discriminati v e information while retaining interpretable g ait characteristics. 4. RESUL TS AND DISCUSSION 4.1. Experimental setup This section e v aluates FusionGait-transformer under a benchmark protocol. The protocol s pecies dataset construction, task denitions, training objecti v es, and e v aluation metrics for g ait identication and g ait- condition classication. Baseline methods are e v aluated under matched input modalities, subject-independent splits, and calibrated training b udgets. Rob ustness analyses co v er speed conditions, vie wpoint changes, and sensor placement perturbations. Statistical reporting is based on repeated runs with dif ferent random seeds and multiple random subject splits. Dataset description: e v aluation uses the CMU motion capture database [47], which pro vides m otion- capture trajectories recorded at 120 Hz for multiple g ait trials per subject. A s u bs et of 120 subjects with balanced gender distrib ution is used. T rials are grouped into slo w , normal, and f ast w alking conditions. V ie wpoint v ariation is represented by dening multiple camera e xtrinsics and projecting 3D joints into 2D pose sequences for each vie w . V irtual IMU signals are generated from the 3D traject ories at anatomically dened lo wer -limb locations using the analytical model of s ection 3.2. Data partitioning follo ws subject-independent splits. Multiple random subject splits and multiple training seeds are used for statistical reporting, and all methods share identical splits to support f air comparison. T raining objecti v es and optimization: training uses a composite objecti v e that combines identity cl as- sication loss, g ait-condition classication loss, and an optional domain-adv ersarial loss implemented with a gradient re v ersal layer , as in (35). L = L id + λ cls L cls + λ dom L dom (35) Hyperparameters λ cls and λ dom are selected on the v alidation set under a x ed tuning b udget shared across methods. Optimizati on uses Adam with init ial learning rate 2 × 10 − 4 , e xponential decay f actor 0.9 e v ery 20 epochs, batch size 32, and dropout 0.3. FusionGait-tr ansformer: multimodal tr ansformer -based fr ame work ... (Kir an K umar S V N Madupu) Evaluation Warning : The document was created with Spire.PDF for Python.