Chelsio Communications Chelsio T5 User manual

froot

Chelsio T5 LIO iSCSI Target Offload Driver for Linux ii
This document and related products are distributed under licenses restricting their use, copying,
distribution, and reverse-engineering.
No part of this document may be reproduced in any form or by any means without prior written permission
by Chelsio Communications.
All third party trademarks are copyright of their respective owners.
THIS DOCUMENTATION IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
THE USE OF THE SOFTWARE AND ANY ASSOCIATED MATERIALS (COLLECTIVELY THE
“SOFTWARE”) IS SUBJECT TO THE SOFTWARE LICENSE TERMS OF CHELSIO
COMMUNICATIONS, INC.
Sales
For all sales inquiries please send email to sa[email protected]
Support
For all support related questions please send email to [e-mail]
Copyright © 2016.Chelsio Communications. All Rights Reserved.
Chelsio ® is a registered trademark of Chelsio Communications.
All other marks and names mentioned herein may be trademarks of their respective companies.
Chelsio Communications (Headquarters)
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A
www.chelsio.com
Tel:[phone]
Fax: [phone]
Chelsio (India) Private Limited
Subramanya Arcade, Floor 3, Tower B
No. 12, Bannerghatta Road,
Bangalore-560029
Karnataka,
India
Tel: [phone]
Chelsio KK (Japan)
Yamato Building 8F,
5-27-3 Sendagaya,
Shibuya-ku,
Tokyo 151-0051,
Japan

Chelsio T5 LIO iSCSI Target Offload Driver for Linux iii
Document History
Version
Revision Date
1.0.0
03/18/2016
1.0.1
03/23/2016
1.0.2
07/29/2016

Chelsio T5 LIO iSCSI Target Offload Driver for Linux iv
TABLE OF CONTENTS
1. Introduction 5
1.1. Features 5
1.2. Hardware Requirements 6
1.3. Software Requirements 6
1.4. Package Contents 7
2. Hardware Installation 8
3. Software/Driver Installation 10
4. Software/Driver Loading 11
5. Software/Driver Configuration and Fine-tuning 12
5.1. Configuring LIO iSCSI Target 12
5.2. Offloading LIO iSCSI Connection 12
5.3. Performance Tuning 13
5.4. Firmware Update 13
6. Software/Driver Unloading 15
7. Software/Driver Update 16
8. Appendix 17
8.1. Chelsio End-User License Agreement (EULA) 17

Chapter 1. Introduction
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 5
1.Introduction
Thank you for choosing Chelsio Unified Wire adapters. These high speed, single chip, single
firmware cards provide enterprises and data centers with high performance solutions for various
Network and Storage related requirements.
The Terminator 5 (T5) is Chelsio’s next generation of highly integrated, hyper-virtualized
40/10GbE controllers. The T5 is built around a programmable protocol-processing engine, with
full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, ISCSI,
FCoE and NAT support. It scales true 40Gb line rate operation from a single TCP connection to
thousands of connections, and allows simultaneous low latency and high bandwidth operation
thanks to multiple physical channels through the ASIC.
The adapters can fully offload TCP, UDP, iSCSI, iWARP and FCoE over a single Unified Wire.
The adapters also fully support SR-IOV, EVB/VNTag, DCB, Traffic Management and Filtering.
Ideal for all data, storage and high performance clustering applications, the adapters enable a
unified fabric over a single wire by simultaneously running all unmodified IP sockets, Fibre
Channel and InfiniBand applications over Ethernet at line rate.
Designed for deployment in virtualized data centers, cloud service installations and high
performance computing environments, Chelsio adapters bring a new level of performance
metrics and functional capabilities to the computer networking industry.
Linux-IO Target (LIO) is the in-kernel SCSI target implementation in Linux. This open-source
standard supports common storage fabrics, including Fibre Channel, FCoE, iEEE 1394, iSCSI,
NVMe-OF, iSER, SRP, USB, vHost, etc. The LIO iSCSI fabric module implements many
advanced iSCSI features that increase performance and resiliency.
This document describes the installation, use and maintenance of the LIO iSCSI Target Offload
Driver and its various components.
1.1. Features
The LIO iSCSI Target Offload Driver provides the following high level features:
Offloads TCP/IP.
Offloads iSCSI Header and Data Digest Calculations.
Offload Speeds at 10Gb and 40Gb.
Supports Direct Data Placement (DDP).
Supports iSCSI Segmentation Offload.
Supports iSCSI PDU recovery.

Chapter 1. Introduction
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 6
1.2. Hardware Requirements
1.2.1. Supported Adapters
The following are the currently shipping Chelsio Adapters that are compatible with LIO iSCSI
Target Offload Driver:
T580-CR
T580-LP-CR
T540-CR
T520-LL-CR
T520-CR
T520-BT
1.3. Software Requirements
cxgb4, iscsi_target_mod, target_core_mod, ipv6 modules are required by LIO iSCSI Target
Offload (cxgbit.ko)module to work.
1.3.1. Linux Requirements
Currently the LIO iSCSI Target Offload Driver is available for the following version(s):
RHEL 7.2, 3.10.0-327.el7
RHEL 7.1, 3.10.0-229.el7
RHEL 7.0, 3.10.0-123.el7
RHEL 6.8, 2.6.32-642.el6
RHEL 6.7, 2.6.32-573.el6
RHEL 6.6, 2.6.32-504.el6
Other versions have not been tested and are not guaranteed to work.
1.3.2. iSCSI Requirements
The LIO iSCSI Target Offload driver requires the following components to be installed to
function:
Python* (v2.7.10 provided in the package)
TargetCLI* (v2.1 provided in the package)
OpenSSL (Download from https://www.openssl.org/source/)
* If not already present in the system, or if an older version exists, the component provided in
the package will be installed along with the kernel.

Chapter 1. Introduction
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 7
1.4. Package Contents
The LIO iSCSI Target Offload Driver package for Linux consists of the following:
scripts: Support script used by the installer.
src: The directory contains T5 firmware binary and configuration, Linux-IO Target kernel and
Python installation files.
utils:
target: The directory contains targetcli installation files and dependent components.
t4_latencytune.sh: Script used for latency tuning of Chelsio Adapters.
t4_perftune.sh: This shell script is to tune the system for higher performance. It
achieves it through modifying the IRQ-CPU binding. This script can also be used to
change Tx coalescing settings.
EULA: Chelsio End User License Agreement.
install.sh: Shell script that installs the kernel module and related components.
docs: The directory contains support documents - README, Release Notes and User’s
Guide (this document) for the software.

Chapter 2. Hardware Installation
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 8
2.Hardware Installation
Follow these steps to install Chelsio Adapter in your system:
1. Shutdown/power off your system.
2. Power off all remaining peripherals attached to your system.
3. Unpack the Chelsio adapter and place it on an anti-static surface.
4. Remove the system case cover according to the system manufacturer’s instructions.
5. Remove the PCI filler plate from the slot where you will install the Ethernet adapter.
6. For maximum performance, it is highly recommended to install the adapter into a PCIe
x8/x16 slot.
7. Holding the Chelsio adapter by the edges, align the edge connector with the PCI connector
on the motherboard. Apply even pressure on both edges until the card is firmly seated. It
may be necessary to remove the SFP (transceiver) modules prior to inserting the adapter.
8. Secure the Chelsio adapter with a screw, or other securing mechanism, as described by the
system manufacturer’s instructions. Replace the case cover.
9. After securing the card, ensure that the card is still fully seated in the PCIE x8 slot as
sometimes the process of securing the card causes the card to become unseated.
10. Connect a fiber cable, multi-mode for short range (SR) optics or single-mode for long range
(LR) optics, to the 40/10Gb Ethernet adapter or regular Ethernet cable for the 1Gb Ethernet
adapter.
11. Power on your system.
12. Run update-pciids command to download the current version of PCI ID list
[root@host~]# update-pciids
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 198k 100 198k 0 0 491k 0 --:--:-- --:--:-- --:--:-- 626k
Done.
13. Verify if the adapter was installed successfully by using the lspci command
[root@host~]# lspci |grep -i Chelsio
07:00.0 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.1 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.2 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.3 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller
07:00.4 Ethernet controller: Chelsio Communications Inc T520-LL-CR Unified
Wire Ethernet Controller

Chapter 2. Hardware Installation
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 9
07:00.5 SCSI storage controller: Chelsio Communications Inc T520-LL-CR
Unified Wire Storage Controller
07:00.6 Fibre Channel: Chelsio Communications Inc T520-LL-CR Unified Wire
Storage Controller
For Chelsio T5 adapters, the physical functions are currently assigned as:
Physical functions 0 - 3: for the SR-IOV functions of the adapter
Physical function 4: for all NIC functions of the adapter
Physical function 5: for iSCSI
Physical function 6: for FCoE
Physical function 7: Currently not assigned
Once driver package is installed and loaded, examine the output of dmesg to see if the card is
discovered.
cxgb4 0000:04:00.4 eth1: Chelsio T520-LL rev 1 1000/10GBASE-R SFP+ RNIC PCIe
x8 5 GT/s MSI-X
cxgb4 0000:04:00.4 eth1: S/N: PT03140249, P/N: 110116750C0
The above outputs indicate the hardware configuration of the adapters as well as the serial
number.
Network device names for Chelsio’s physical ports are assigned using the following
convention: the port farthest from the motherboard will appear as the first network
interface. However, for T5 40G adapters, the association of physical Ethernet ports
and their corresponding network device names is opposite. For these adapters, the
port nearest to the motherboard will appear as the first network interface.
Note

Chapter 3. Software/Driver Installation
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 10
3.Software/Driver Installation
i. Download the LIO iSCSI Target Offload driver package from Chelsio Download Center,
http://service.chelsio.com/
ii. Untar the tarball using the following command:
[root@host~]# tar zxvfm Chelsio-LIO-iSCSITarget-offload-x.x.x.x.tar.gz
iii. Change your current working directory to Chelsio-LIO-iSCSITarget-offload-x.x.x.x directory
and run the following script to start the installation:
[root@host~]# ./install.sh -t
iv. Reboot system into the newly installed 4.6.4 kernel.

Chapter 4. Software/Driver Loading
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 11
4.Software/Driver Loading
Load the LIO iSCSI Target Offload driver (cxgbit) using:
[root@host~]# modprobe cxgbit

Chapter 5. Software/Driver Configuration and Fine-tuning
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 12
5.Software/Driver Configuration and Fine-tuning
5.1. Configuring LIO iSCSI Target
The LIO iSCSI Target needs to be configured before it can become useful.
Please refer the user manual at http://linux-iscsi.org/builds/docs/LIO-3.0-
Users_Reference_Manual.pdf to do so.
5.1.1. Sample Configuration
Here is a sample iSCSI configuration listing a target configured with 1 RAM disk LUN and ACL
not configured:
[root@host~]# targetcli ls
o- / .........................................................[...]
o- backstores.....................................................[...]
| o- block..........................................[Storage Objects: 0]
| o- fileio .....................................[Storage Objects: 0]
| o- pscsi .......................................[Storage Objects: 0]
| o- ramdisk ........................................[Storage Objects: 1]
| | o- ramdisk01 ...................................[(300.0MiB) activated]
| o- user ........................................[Storage Objects: 0]
o- iscsi ...........................................[Targets: 1]
| o- iqn.2015-12.org.linux-iscsi.proton.target01...................[TPGs: 1]
| o- tpg1 .........................................[gen-acls, no-auth]
| o- acls ...............................................[ACLs: 0]
| o- luns ...............................................[LUNs: 1]
| | o- lun1 .......................................[ramdisk/ramdisk01]
| o- portals ..........................................[Portals: 1]
| o- 102.10.10.121:3260 .........................................[OK]
o- loopback ...............................................[Targets: 0]
o- vhost ................................................[Targets: 0]
5.2. Offloading LIO iSCSI Connection
To offload the LIO iSCSI Target use the following command:
[root@host~]# echo 1 >
/sys/kernel/config/target/iscsi/<target_iqn>/tpgt_1/np/<target_ip>\:3260/cxg
bit

Chapter 5. Software/Driver Configuration and Fine-tuning
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 13
E.g.:
[root@host~]# echo 1 > /sys/kernel/config/target/iscsi/iqn.2015-
12.org.linux-iscsi.proton.target01/tpgt_1/np/102.20.20.241\:3260/cxgbit
Execute the above command for every portal address listening on Chelsio interface.
5.3. Performance Tuning
In order to auto tune the system for best performance, Chelsio recommends:
Disabling virtualization, c-state technology, VT-d, Intel I/O AT and SR-IOV in the BIOS
settings.
Installing the adapter into a PCIe Gen3 x8/x16 slot.
Disable SELinux and firewall.
Installing the driver package will copy t4_perftune.sh script to /sbin directory. Run the
script to map the adapter queues to different CPUs:
[root@host~]# t4_perftune.sh
Also, follow the steps mentioned below to lower your latency:
i. Run the following script to disable few services.
[root@host~]# t4_latencytune.sh <interface>
ii. Set sysctl param net.ipv4.tcp_low_latency to 1
[root@host~]# sysctl -w net.ipv4.tcp_low_latency=1
5.4. Firmware Update
The T5 firmware binary and configruation file are installed on the system, typically in
/lib/firmware/cxgb4, and the driver will auto-load the firmware if an update is required. The
kernel must be configured to enable userspace firmware loading support:
Device Drivers -> Generic Driver Options -> Userspace firmware loading support

Chapter 5. Software/Driver Configuration and Fine-tuning
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 14
The firmware version can be verified using ethtool:
[root@host~]# ethtool -i <iface>
In case the adapter has a higher firmware version, you will have to downgrade using the
following command:
[root@host~]# ethtool -f ethX cxgb4/t5fw-1.15.37.0.bin
Reboot system for changes to take effect.

Chapter 6. Software/Driver Unloading
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 15
6.Software/Driver Unloading
To unload the LIO iSCSI Target Offload kernel module, follow the steps mentioned below:
i. Log out from the initiator.
ii. Run the following command:
[root@host~]# echo 0 >
/sys/kernel/config/target/iscsi/<target_iqn>/tpgt_1/np/<target_ip>\:3260/cxg
bit
Execute the above command for every portal address listening on Chelsio interface.
iii. Unload the driver:
[root@host~]# rmmod cxgbit

Chapter 7. Software/Driver Update
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 16
7.Software/Driver Update
For any distribution specific problems, please check README and Release Notes included in
the release for possible workaround.
Please visit Chelsio support web site http://service.chelsio.com/ for regular updates on various
software/drivers. You can also subscribe to our newsletter for the latest software updates.

Chapter 8. Appendix
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 17
8.Appendix
8.1. Chelsio End-User License Agreement (EULA)
Installation and use of the driver/software implies acceptance of the terms in the Chelsio End-
User License Agreement (EULA).
IMPORTANT: PLEASE READ THIS SOFTWARE LICENSE CAREFULLY BEFORE DOWNLOADING OR OTHERWISE
USING THE SOFTWARE OR ANY ASSOCIATED DOCUMENTATION OR OTHER MATERIALS
(COLLECTIVELY, THE "SOFTWARE"). BY CLICKING ON THE "OK" OR "ACCEPT" BUTTON YOU
AGREE TO BE BOUND BY THE TERMS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO THE TERMS OF
THIS AGREEMENT, CLICK THE "DO NOT ACCEPT" BUTTON TO TERMINATE THE INSTALLATION
PROCESS.
1. License. Chelsio Communications, Inc. ("Chelsio") hereby grants you, the Licensee,
and you hereby accept, a limited, non-exclusive, non-transferable license to install
and use the Software with one or more Chelsio network adapters on a single server
computer for use in communicating with one or more other computers over a network.
You may also make one copy of the Software in machine readable form solely for back-up
purposes, provided you reproduce Chelsio's copyright notice and any proprietary
legends included with the Software or as otherwise required by Chelsio.
2. Restrictions. This license granted hereunder does not constitute a sale of the
Software or any copy thereof. Except as expressly permitted under this Agreement, you
may not:
(i) reproduce, modify, adapt, translate, rent, lease, loan, resell, distribute, or
create derivative works of or based upon, the Software or any part thereof; or
(ii) make available the Software, or any portion thereof, in any form, on the
Internet. The Software contains trade secrets and, in order to protect them, you may
not decompile, reverse engineer, disassemble, or otherwise reduce the Software to a
human-perceivable form. You assume full responsibility for the use of the Software and
agree to use the Software legally and responsibly.
3. Ownership of Software. As Licensee, you own only the media upon which the Software
is recorded or fixed, but Chelsio retains all right, title and interest in and to the
Software and all subsequent copies of the Software, regardless of the form or media in
or on which the Software may be embedded.
4. Confidentiality. You agree to maintain the Software in confidence and not to
disclose the Software, or any information or materials related thereto, to any third
party without the express written consent of Chelsio. You further agree to take all
reasonable precautions to limit access of the Software only to those of your employees
who reasonably require such access to perform their employment obligations and who are
bound by confidentiality agreements with you.
5. Term. This license is effective in perpetuity, unless terminated earlier. You may
terminate the license at any time by destroying the Software (including the related
documentation), together with all copies or modifications in any form. Chelsio may
terminate this license, and this license shall be deemed to have automatically
terminated, if you fail to comply with any term or condition of this Agreement. Upon
any termination, including termination by you, you must destroy the Software

Chapter 8. Appendix
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 18
(including the related documentation), together with all copies or modifications in
any form.
6. Limited Warranty. If Chelsio furnishes the Software to you on media, Chelsio
warrants only that the media upon which the Software is furnished will be free from
defects in material or workmanship under normal use and service for a period of thirty
(30) days from the date of delivery to you.
CHELSIO DOES NOT AND CANNOT WARRANT THE PERFORMANCE OR RESULTS YOU MAY OBTAIN BY USING
THE SOFTWARE OR ANY PART THEREOF. EXCEPT FOR THE FOREGOING LIMITED WARRANTY, CHELSIO
MAKES NO OTHER WARRANTIES, EXPRESS OR IMPLIED, AND HEREBY DISCLAIMS ALL OTHER
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, NON-INFRINGEMENT OF THIRD PARTY RIGHTS,
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow the
exclusion of implied warranties or limitations on how long an implied warranty may
last, so the above limitations may not apply to you. This warranty gives you specific
legal rights and you may also have other rights which vary from state to state.
7. Remedy for Breach of Warranty. The sole and exclusive liability of Chelsio and its
distributors, and your sole and exclusive remedy, for a breach of the above warranty,
shall be the replacement of any media furnished by Chelsio not meeting the above
limited warranty and which is returned to Chelsio. If Chelsio or its distributor is
unable to deliver replacement media which is free from defects in materials or
workmanship, you may terminate this Agreement by returning the Software.
8. Limitation of Liability. IN NO EVENT SHALL CHELSIO HAVE ANY LIABILITY TO YOU OR ANY
THIRD PARTY FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR PUNITIVE
DAMAGES, HOWEVER CAUSED, AND ON ANY THEORY OF LIABILITY, ARISING OUT OF OR RELATED TO
THE LICENSE OR USE OF THE SOFTWARE, INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR LOSS
OF ANTICIPATED PROFITS, EVEN IF CHELSIO HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES. IN NO EVENT SHALL CHELSIO'S LIABILITY ARISING OUT OF OR RELATED TO THE
LICENSE OR USE OF THE SOFTWARE EXCEED THE AMOUNTS PAID BY YOU FOR THE LICENSE GRANTED
HEREUNDER. THESE LIMITATIONS SHALL APPLY NOTWITHSTANDING ANY FAILURE OF ESSENTIAL
PURPOSE OF ANY LIMITED REMEDY.
9. High Risk Activities. The Software is not fault-tolerant and is not designed,
manufactured or intended for use or resale as online equipment control equipment in
hazardous environments requiring fail-safe performance, such as in the operation of
nuclear facilities, aircraft navigation or communication systems, air traffic control,
direct life support machines, or weapons systems, in which the failure of the Software
could lead directly to death, personal injury, or severe physical or environmental
damage. Chelsio specifically disclaims any express or implied warranty of fitness for
any high risk uses listed above.
10. Export. You acknowledge that the Software is of U.S. origin and subject to U.S.
export jurisdiction. You acknowledge that the laws and regulations of the United
States and other countries may restrict the export and re-export of the Software. You
agree that you will not export or re-export the Software or documentation in any form
in violation of applicable United States and foreign law. You agree to comply with all
applicable international and national laws that apply to the Software, including the
U.S.
Export Administration Regulations, as well as end-user, end-use, and destination
restrictions issued by U.S. and other governments.
11. Government Restricted Rights. The Software is subject to restricted rights as
follows. If the Software is acquired under the terms of a GSA contract: use,

Chapter 8. Appendix
Chelsio T5 LIO iSCSI Target Offload Driver for Linux 19
reproduction or disclosure is subject to the restrictions set forth in the applicable
ADP Schedule contract. If the Software is acquired under the terms of a DoD or
civilian agency contract, use, duplication or disclosure by the Government is subject
to the restrictions of this Agreement in accordance with 48 C.F.R. 12.212 of the
Federal Acquisition Regulations and its successors and 49 C.F.R. 227.7202-1 of the DoD
FAR Supplement and its successors.
12. General. You acknowledge that you have read this Agreement, understand it, and
that by using the Software you agree to be bound by its terms and conditions. You
further agree that it is the complete and exclusive statement of the agreement between
Chelsio and you, and supersedes any proposal or prior agreement, oral or written, and
any other communication between Chelsio and you relating to the subject matter of this
Agreement. No additional or any different terms will be enforceable against Chelsio
unless Chelsio gives its express consent, including an express waiver of the terms of
this Agreement, in writing signed by an officer of Chelsio. This Agreement shall be
governed by California law, except as to copyright matters, which are covered by
Federal law. You hereby irrevocably submit to the personal jurisdiction of, and
irrevocably waive objection to the laying of venue (including a waiver of any argument
of forum non conveniens or other principles of like effect) in, the state and federal
courts located in Santa Clara County, California, for the purposes of any litigation
undertaken in connection with this Agreement. Should any provision of this Agreement
be declared unenforceable in any jurisdiction, then such provision shall be deemed
severable from this Agreement and shall not affect the remainder hereof. All rights in
the Software not specifically granted in this Agreement are reserved by Chelsio. You
may not assign or transfer this Agreement (by merger, operation of law or in any
other manner) without the prior written consent of Chelsio and any attempt to do so
without such consent shall be void and shall constitute a material breach of this
Agreement.
Should you have any questions concerning this Agreement, you may contact Chelsio by
writing to:
Chelsio Communications, Inc.
209 North Fair Oaks Avenue,
Sunnyvale, CA 94085
U.S.A